From 1710e48c508e5e7d0432fb7b431a2538b1444d66 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 14:10:50 +0800 Subject: [PATCH 01/64] Add Groovy replacement plan for MAL, LAL, and Hierarchy scripts Document the detailed implementation plan for eliminating Groovy from OAP runtime via build-time transpilers (MAL/LAL) and v1/v2 module split (hierarchy), based on Discussion #13716 and skywalking-graalvm-distro. Co-Authored-By: Claude Opus 4.6 --- docs/en/academy/groovy-replacement-plan.md | 912 +++++++++++++++++++++ 1 file changed, 912 insertions(+) create mode 100644 docs/en/academy/groovy-replacement-plan.md diff --git a/docs/en/academy/groovy-replacement-plan.md b/docs/en/academy/groovy-replacement-plan.md new file mode 100644 index 000000000000..101861503815 --- /dev/null +++ b/docs/en/academy/groovy-replacement-plan.md @@ -0,0 +1,912 @@ +# Groovy Replacement Plan: Build-Time Transpiler for MAL, LAL, and Hierarchy Scripts + +Reference: [Discussion #13716](https://github.com/apache/skywalking/discussions/13716) +Reference Implementation: [skywalking-graalvm-distro](https://github.com/apache/skywalking-graalvm-distro) + +## 1. Background and Motivation + +SkyWalking OAP server currently uses Groovy as the runtime scripting engine for three subsystems: + +| Subsystem | YAML Files | Expressions | Groovy Pattern | +|-----------|-----------|-------------|----------------| +| MAL (Meter Analysis Language) | 71 (11 meter-analyzer-config, 55 otel-rules, 2 log-mal-rules, 2 envoy-metrics-rules, 1 telegraf-rules) | 1,254 metric + 29 filter | Dynamic Groovy: `propertyMissing()`, `ExpandoMetaClass` on `Number`, closures | +| LAL (Log Analysis Language) | 8 | 10 rules | `@CompileStatic` Groovy: delegation-based closure DSL, safe navigation (`?.`), `as` casts | +| Hierarchy Matching | 1 (hierarchy-definition.yml) | 4 rules | `GroovyShell.evaluate()` for `Closure` | + +### Problems with Groovy Runtime + +1. **Startup Cost**: 1,250+ `GroovyShell.parse()` calls at OAP boot, each spinning up the full Groovy compiler pipeline. +2. **Runtime Errors Instead of Compile-Time Errors**: MAL uses dynamic Groovy -- typos in metric names or invalid method chains are only discovered when that specific expression runs with real data. +3. **Debugging Complexity**: Stack traces include Groovy MOP internals (`CallSite`, `MetaClassImpl`, `ExpandoMetaClass`), obscuring the actual expression logic. +4. **Runtime Execution Performance (Most Critical)**: MAL expressions execute on every metrics ingestion cycle. Per-expression overhead from dynamic Groovy compounds at scale: + - Property resolution: `CallSite` -> `MetaClassImpl.invokePropertyOrMissing()` -> `ExpressionDelegate.propertyMissing()` -> `ThreadLocal` lookup (4+ layers of indirection per metric name lookup) + - Method calls: Groovy `CallSite` dispatch with MetaClass lookup and MOP interception checks + - Arithmetic (`metric * 1000`): `ExpandoMetaClass` closure allocation + metaclass lookup + dynamic dispatch for what Java does as a single `imul` + - Per ingestion cycle: ~1,250 `propertyMissing()` calls, ~3,750 MOP method dispatches, ~29 metaclass arithmetic ops, ~200 closure allocations + - JIT cannot optimize Groovy's megamorphic call sites, defeating inlining and branch prediction +5. **GraalVM Incompatibility**: `invokedynamic` bootstrapping and `ExpandoMetaClass` are fundamentally incompatible with AOT compilation. + +### Goal + +Eliminate Groovy from the OAP runtime entirely. Groovy becomes a **build-time-only** dependency used solely for AST parsing by the transpiler. Zero `GroovyShell`, zero `ExpandoMetaClass`, zero MOP at runtime. + +--- + +## 2. Solution Architecture + +### Build-Time Transpiler: Groovy DSL -> Pure Java Source Code + +``` +BUILD TIME (Maven compile phase): + MAL YAML files (71 files, 1,250+ expressions) + LAL YAML files (8 files, 10 scripts) + | + v + MalToJavaTranspiler / LalToJavaTranspiler + (Groovy CompilationUnit at CONVERSION phase -- AST parsing only, no execution) + | + v + ~1,254 MalExpr_*.java + ~6 LalExpr_*.java + MalFilter_*.java + | + v + javax.tools.JavaCompiler -> .class files on classpath + META-INF/mal-expressions.txt (manifest) + META-INF/mal-filter-expressions.properties (manifest) + META-INF/lal-expressions.txt (manifest) + +RUNTIME (OAP Server): + Class.forName(className) -> MalExpression / LalExpression instance + Zero Groovy. Zero GroovyShell. Zero ExpandoMetaClass. +``` + +The transpiler approach is already fully implemented and validated in the [skywalking-graalvm-distro](https://github.com/apache/skywalking-graalvm-distro) repository. + +--- + +## 3. Detailed Design + +### 3.1 New Functional Interfaces + +Three core interfaces replace Groovy's `DelegatingScript` and `Closure`: + +```java +// MAL: replaces DelegatingScript + ExpandoMetaClass + ExpressionDelegate.propertyMissing() +@FunctionalInterface +public interface MalExpression { + SampleFamily run(Map samples); +} + +// MAL: replaces Closure from GroovyShell.evaluate() for filter expressions +@FunctionalInterface +public interface MalFilter { + boolean test(Map tags); +} + +// LAL: replaces LALDelegatingScript + @CompileStatic closure DSL +@FunctionalInterface +public interface LalExpression { + void execute(FilterSpec filterSpec, Binding binding); +} +``` + +### 3.2 SampleFamily: Closure -> Functional Interface + +Five `SampleFamily` methods currently accept `groovy.lang.Closure`. Each gets a new overload with a Java functional interface. During transition both overloads coexist; eventually the Closure overloads are removed. + +| Method | Current (Groovy) | New (Java) | Functional Interface | +|--------|-----------------|------------|---------------------| +| `tag()` | `Closure` with `tags.key = val` | `TagFunction` | `Function, Map>` | +| `filter()` | `Closure` with `tags.x == 'y'` | `SampleFilter` | `Predicate>` | +| `forEach()` | `Closure` with `(prefix, tags) -> ...` | `ForEachFunction` | `BiConsumer>` | +| `decorate()` | `Closure` with `entity -> ...` | `DecorateFunction` | `Consumer` | +| `instance(..., closure)` | `Closure` with `tags -> Map.of(...)` | `PropertiesExtractor` | `Function, Map>` | + +Source location in upstream: `oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java` + +### 3.3 MAL Transpiler: AST Mapping Rules + +The `MalToJavaTranspiler` (~1,230 lines) parses each MAL expression string into a Groovy AST at `Phases.CONVERSION` (no code execution), then walks the AST to emit equivalent Java code. + +#### Expression Mappings + +| Groovy Construct | Java Output | Notes | +|-----------------|-------------|-------| +| `metric_name` (bare property) | `samples.getOrDefault("metric_name", SampleFamily.EMPTY)` | Replaces `propertyMissing()` dispatch | +| `.sum(['a','b'])` | `.sum(List.of("a", "b"))` | Direct method call | +| `.tagEqual('resource', 'cpu')` | `.tagEqual("resource", "cpu")` | Direct method call | +| `100 * metric` | `metric.multiply(100)` | Commutative: operands swapped | +| `100 - metric` | `metric.minus(100).negative()` | Non-commutative: negate | +| `100 / metric` | `metric.newValue(v -> 100 / v)` | Non-commutative: newValue | +| `metricA / metricB` | `metricA.div(metricB)` | SampleFamily-SampleFamily op | +| `.tag({tags -> tags.cluster = ...})` | `.tag(tags -> { tags.put("cluster", ...); return tags; })` | Closure -> lambda | +| `.filter({tags -> tags.job_name in [...]})` | `.filter(tags -> "...".equals(tags.get("job_name")))` | Closure -> predicate | +| `.forEach(['a','b'], {p, tags -> ...})` | `.forEach(List.of("a","b"), (p, tags) -> { ... })` | Closure -> BiConsumer | +| `.decorate({entity -> ...})` | `.decorate(entity -> { ... })` | Closure -> Consumer | +| `.instance(..., {tags -> Map.of(...)})` | `.instance(..., tags -> Map.of(...))` | Closure -> Function | +| `Layer.K8S` | `Layer.K8S` | Enum constant, passed through | +| `time()` | `Instant.now().getEpochSecond()` | Direct Java API | +| `AVG`, `SUM`, etc. | `DownsamplingType.AVG`, etc. | Enum constant reference | + +#### Closure Body Translation + +Inside closures, Groovy property-style access is mapped to explicit `Map` operations: + +| Groovy Closure Pattern | Java Lambda Output | +|----------------------|-------------------| +| `tags.key = "value"` | `tags.put("key", "value")` | +| `tags.key` (read) | `tags.get("key")` | +| `tags.remove("key")` | `tags.remove("key")` | +| `tags.key == "value"` | `"value".equals(tags.get("key"))` | +| `tags.key != "value"` | `!"value".equals(tags.get("key"))` | +| `tags.key in ["a","b"]` | `List.of("a","b").contains(tags.get("key"))` | +| `if/else` in closure | `if/else` in lambda | +| `entity.serviceId = val` | `entity.setServiceId(val)` | + +#### Filter Expression Mappings + +Filter expressions (`filter: "{ tags -> ... }"`) generate `MalFilter` implementations: + +| Groovy Filter | Java Output | +|--------------|-------------| +| `tags.job_name == 'mysql'` | `"mysql".equals(tags.get("job_name"))` | +| `tags.job_name != 'test'` | `!"test".equals(tags.get("job_name"))` | +| `tags.job_name in ['a','b']` | `List.of("a","b").contains(tags.get("job_name"))` | +| `cond1 && cond2` | `cond1 && cond2` | +| `cond1 \|\| cond2` | `cond1 \|\| cond2` | +| `!cond` | `!cond` | +| `tags.job_name` (truthiness) | `tags.get("job_name") != null` | + +### 3.4 LAL Transpiler: AST Mapping Rules + +The `LalToJavaTranspiler` (~950 lines) handles LAL's `@CompileStatic` delegation-based DSL. LAL scripts have a fundamentally different structure from MAL -- they are statement-based builder patterns rather than expression-based computations. + +#### Statement Mappings + +| Groovy Construct | Java Output | +|-----------------|-------------| +| `filter { ... }` | Body unwrapped, emitted directly on `filterSpec` | +| `json {}` | `filterSpec.json()` | +| `json { abortOnFailure false }` | `filterSpec.json(jp -> { jp.abortOnFailure(false); })` | +| `text { regexp /pattern/ }` | `filterSpec.text(tp -> { tp.regexp("pattern"); })` | +| `yaml {}` | `filterSpec.yaml()` | +| `extractor { ... }` | `filterSpec.extractor(ext -> { ... })` | +| `sink { ... }` | `filterSpec.sink(s -> { ... })` | +| `abort {}` | `filterSpec.abort()` | +| `service parsed.service as String` | `ext.service(String.valueOf(getAt(binding.parsed(), "service")))` | +| `layer parsed.layer as String` | `ext.layer(String.valueOf(getAt(binding.parsed(), "layer")))` | +| `tag(key: val)` | `ext.tag(Map.of("key", val))` | +| `timestamp parsed.time as String` | `ext.timestamp(String.valueOf(getAt(binding.parsed(), "time")))` | + +#### Property Access and Safe Navigation + +| Groovy Pattern | Java Output | +|---------------|-------------| +| `parsed.field` | `getAt(binding.parsed(), "field")` | +| `parsed.field.nested` | `getAt(getAt(binding.parsed(), "field"), "nested")` | +| `parsed?.field?.nested` | `((__v0 = binding.parsed()) == null ? null : ((__v1 = getAt(__v0, "field")) == null ? null : getAt(__v1, "nested")))` | +| `log.tags` | `binding.log().getTags()` | + +#### Cast and Type Handling + +| Groovy Pattern | Java Output | +|---------------|-------------| +| `expr as String` | `String.valueOf(expr)` | +| `expr as Long` | `toLong(expr)` | +| `expr as Integer` | `toInt(expr)` | +| `expr as Boolean` | `toBoolean(expr)` | +| `"${expr}"` (GString) | `"" + expr` | + +#### LAL Spec Consumer Overloads + +LAL spec classes (`FilterSpec`, `ExtractorSpec`, `SinkSpec`) get additional method overloads accepting `java.util.function.Consumer` alongside existing Groovy `Closure` parameters: + +```java +// FilterSpec - existing +public void extractor(Closure cl) { ... } +// FilterSpec - new overload +public void extractor(Consumer consumer) { ... } + +// SinkSpec - existing +public void sampler(Closure cl) { ... } +// SinkSpec - new overload +public void sampler(Consumer consumer) { ... } +``` + +Methods requiring Consumer overloads: `text()`, `json()`, `yaml()`, `extractor()`, `sink()`, `slowSql()`, `sampledTrace()`, `metrics()`, `sampler()`, `enforcer()`, `dropper()`. + +#### SHA-256 Deduplication + +LAL manifest is keyed by SHA-256 hash of the DSL content. Identical scripts across different YAML files share one compiled class. In practice, 10 LAL rules map to 6 unique classes. + +### 3.5 Hierarchy Script: v1/v2 Module Split + +The hierarchy matching rules in `hierarchy-definition.yml` use `GroovyShell.evaluate()` to compile 4 Groovy closures at runtime. Unlike MAL/LAL, hierarchy does not need a transpiler (only 4 rules, finite set), but it follows the same v1/v2/checker module pattern for consistency and to remove Groovy from `server-core`. + +#### Current State (server-core, Groovy-coupled) + +`HierarchyDefinitionService.java` lives in `server-core` and is registered as a `Service` in `CoreModule`. Its inner class `MatchingRule` holds a Groovy `Closure`: + +```java +// server-core/...config/HierarchyDefinitionService.java (current) +public static class MatchingRule { + private final String name; + private final String expression; + private final Closure closure; // groovy.lang.Closure + + public MatchingRule(final String name, final String expression) { + GroovyShell sh = new GroovyShell(); + closure = (Closure) sh.evaluate(expression); // Groovy at runtime + } +} +``` + +This `MatchingRule` is referenced by three classes in `server-core`: +- `HierarchyDefinitionService` -- builds the rule map from YAML +- `HierarchyService` -- calls `matchingRule.getClosure().call(service, comparedService)` for auto-matching +- `HierarchyQueryService` -- reads the hierarchy definition map + +#### Step 1: Make server-core Groovy-Free + +Refactor `MatchingRule` in `server-core` to use a Java functional interface instead of `Closure`: + +```java +// server-core/...config/HierarchyDefinitionService.java (refactored) +public static class MatchingRule { + private final String name; + private final String expression; + private final BiFunction matcher; // pure Java + + public MatchingRule(final String name, final String expression, + final BiFunction matcher) { + this.name = name; + this.expression = expression; + this.matcher = matcher; + } + + public boolean match(Service upper, Service lower) { + return matcher.apply(upper, lower); + } +} +``` + +`HierarchyDefinitionService.init()` no longer compiles Groovy expressions itself. Instead, it receives a `Map>` (the rule registry) from outside -- injected by whichever implementation module (v1 or v2) is active. + +`HierarchyService` changes from `matchingRule.getClosure().call(u, l)` to `matchingRule.match(u, l)`. + +Remove all `groovy.lang.*` imports from `server-core`. + +#### Step 2: hierarchy-v1 (Groovy-based, for checker only) + +```java +// analyzer/hierarchy-v1/.../GroovyHierarchyRuleProvider.java +public class GroovyHierarchyRuleProvider { + public static Map> buildRules( + Map ruleExpressions) { + Map> rules = new HashMap<>(); + GroovyShell sh = new GroovyShell(); + ruleExpressions.forEach((name, expression) -> { + Closure closure = (Closure) sh.evaluate(expression); + rules.put(name, (u, l) -> closure.call(u, l)); + }); + return rules; + } +} +``` + +This module depends on Groovy and wraps the original `GroovyShell.evaluate()` logic. It is NOT included in the runtime classpath -- only used by the checker. + +#### Step 3: hierarchy-v2 (Pure Java, for runtime) + +```java +// analyzer/hierarchy-v2/.../JavaHierarchyRuleProvider.java +public class JavaHierarchyRuleProvider { + private static final Map> RULE_REGISTRY; + static { + RULE_REGISTRY = new HashMap<>(); + RULE_REGISTRY.put("name", + (u, l) -> Objects.equals(u.getName(), l.getName())); + RULE_REGISTRY.put("short-name", + (u, l) -> Objects.equals(u.getShortName(), l.getShortName())); + RULE_REGISTRY.put("lower-short-name-remove-ns", (u, l) -> { + String sn = l.getShortName(); + int dot = sn.lastIndexOf('.'); + return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); + }); + RULE_REGISTRY.put("lower-short-name-with-fqdn", (u, l) -> { + String sn = u.getShortName(); + int colon = sn.lastIndexOf(':'); + return colon > 0 && Objects.equals( + sn.substring(0, colon), + l.getShortName() + ".svc.cluster.local"); + }); + } + + public static Map> buildRules( + Map ruleExpressions) { + Map> rules = new HashMap<>(); + ruleExpressions.forEach((name, expression) -> { + BiFunction fn = RULE_REGISTRY.get(name); + if (fn == null) { + throw new IllegalArgumentException( + "Unknown hierarchy matching rule: " + name + + ". Known rules: " + RULE_REGISTRY.keySet()); + } + rules.put(name, fn); + }); + return rules; + } +} +``` + +Unknown rule names fail fast at startup with `IllegalArgumentException`. The YAML file (`hierarchy-definition.yml`) continues to reference rule names (`name`, `short-name`, etc.) -- the Groovy expression strings in `auto-matching-rules` become documentation-only at runtime. + +#### Step 4: hierarchy-v1-v2-checker + +```java +// analyzer/hierarchy-v1-v2-checker/.../HierarchyRuleComparisonTest.java +class HierarchyRuleComparisonTest { + // Load rule expressions from hierarchy-definition.yml + // For each rule: + // Path A: GroovyHierarchyRuleProvider.buildRules() (v1) + // Path B: JavaHierarchyRuleProvider.buildRules() (v2) + // Construct test Service pairs (matching and non-matching cases) + // Assert v1.match(u, l) == v2.match(u, l) for all test pairs +} +``` + +Test cases cover all 4 rules with realistic service name patterns: +- `name`: exact match and mismatch +- `short-name`: exact shortName match and mismatch +- `lower-short-name-remove-ns`: `"svc" == "svc.namespace"` and edge cases (no dot, empty) +- `lower-short-name-with-fqdn`: `"db:3306"` vs `"db.svc.cluster.local"` and edge cases (no colon, wrong suffix) + +--- + +## 4. Module Structure + +### 4.1 Upstream Module Layout + +``` +oap-server/ + server-core/ # MODIFIED: MatchingRule uses BiFunction (no Groovy imports) + + analyzer/ + meter-analyzer/ # Modified: add MalExpression, functional interfaces + log-analyzer/ # Modified: add LalExpression, Consumer overloads + + mal-lal-v1/ # NEW: Move existing Groovy-based code here + meter-analyzer-v1/ # Original MAL (GroovyShell + ExpandoMetaClass) + log-analyzer-v1/ # Original LAL (GroovyShell + @CompileStatic) + + mal-lal-v2/ # NEW: Pure Java transpiler-based implementations + meter-analyzer-v2/ # MalExpression loader + functional interface dispatch + log-analyzer-v2/ # LalExpression loader + Consumer dispatch + mal-transpiler/ # Build-time: Groovy AST -> Java source (MAL) + lal-transpiler/ # Build-time: Groovy AST -> Java source (LAL) + + mal-lal-v1-v2-checker/ # NEW: Dual-path comparison tests (MAL + LAL) + 73 MAL test classes (1,281 assertions) + 5 LAL test classes (19 assertions) + + hierarchy-v1/ # NEW: Groovy-based hierarchy rule provider (checker only) + hierarchy-v2/ # NEW: Pure Java hierarchy rule provider (runtime) + hierarchy-v1-v2-checker/ # NEW: Dual-path comparison tests (hierarchy) +``` + +### 4.2 Dependency Graph + +``` +mal-transpiler ──────────────> groovy (build-time only, for AST parsing) +lal-transpiler ──────────────> groovy (build-time only, for AST parsing) +hierarchy-v1 ────────────────> groovy (checker only, not runtime) + +meter-analyzer-v2 ──────────> meter-analyzer (interfaces + SampleFamily) +log-analyzer-v2 ────────────> log-analyzer (interfaces + spec classes) +hierarchy-v2 ───────────────> server-core (MatchingRule with BiFunction) + +mal-lal-v1-v2-checker ──────> mal-lal-v1 (Groovy path) +mal-lal-v1-v2-checker ──────> mal-lal-v2 (Java path) +hierarchy-v1-v2-checker ────> hierarchy-v1 (Groovy path) +hierarchy-v1-v2-checker ────> hierarchy-v2 (Java path) + +server-starter ─────────────> meter-analyzer-v2 (runtime, no Groovy) +server-starter ─────────────> log-analyzer-v2 (runtime, no Groovy) +server-starter ─────────────> hierarchy-v2 (runtime, no Groovy) +server-starter ────────────X─> mal-lal-v1 (NOT in runtime) +server-starter ────────────X─> hierarchy-v1 (NOT in runtime) +``` + +### 4.3 Key Design Principle: No Coexistence + +v1 (Groovy) and v2 (Java) never coexist in the OAP runtime classpath. The `mal-lal-v1` and `hierarchy-v1` modules are only dependencies of their respective checker modules for CI validation. The runtime (`server-starter`) depends only on v2 modules. + +--- + +## 5. Implementation Steps + +### Phase 1: Interfaces and SampleFamily Modifications + +**Files to modify:** + +1. **Create `MalExpression.java`** in `meter-analyzer` + - Path: `oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java` + +2. **Create `MalFilter.java`** in `meter-analyzer` + - Path: `oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java` + +3. **Create functional interfaces** in `meter-analyzer` + - `TagFunction extends Function, Map>` + - `SampleFilter extends Predicate>` + - `ForEachFunction extends BiConsumer>` + - `DecorateFunction extends Consumer` + - `PropertiesExtractor extends Function, Map>` + +4. **Add overloads to `SampleFamily.java`** + - Add `tag(TagFunction)` alongside existing `tag(Closure)` + - Add `filter(SampleFilter)` alongside existing `filter(Closure)` + - Add `forEach(List, ForEachFunction)` alongside existing `forEach(List, Closure)` + - Add `decorate(DecorateFunction)` alongside existing `decorate(Closure)` + - Add `instance(..., PropertiesExtractor)` alongside existing `instance(..., Closure)` + +5. **Create `LalExpression.java`** in `log-analyzer` + - Path: `oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java` + +6. **Add Consumer overloads to LAL spec classes** + - `FilterSpec`: `text(Consumer)`, `json(Consumer)`, `yaml(Consumer)`, `extractor(Consumer)`, `sink(Consumer)`, `filter(Consumer)` + - `ExtractorSpec`: `slowSql(Consumer)`, `sampledTrace(Consumer)`, `metrics(Consumer)` + - `SinkSpec`: `sampler(Consumer)`, `enforcer(Consumer)`, `dropper(Consumer)` + +### Phase 2: MAL Transpiler + +**New module: `oap-server/analyzer/mal-lal-v2/mal-transpiler/`** + +1. **`MalToJavaTranspiler.java`** (~1,230 lines) + - Uses `org.codehaus.groovy.control.CompilationUnit` at `Phases.CONVERSION` + - Walks AST via recursive visitor pattern + - Core methods: + - `transpileExpression(String className, String expression)` -> generates Java source for `MalExpression` + - `transpileFilter(String className, String filterLiteral)` -> generates Java source for `MalFilter` + - `collectSampleNames(Expression expr)` -> extracts all metric name references + - `visitExpression(Expression node)` -> recursive Java code emitter + - `visitClosureExpression(ClosureExpression node, String contextType)` -> closure-to-lambda + - `compileAll()` -> batch `javac` compile + manifest generation + +2. **Maven integration**: `exec-maven-plugin` during `generate-sources` phase + - Reads all MAL YAML files from resources + - Generates Java source to `target/generated-sources/mal/` + - Compiles to `target/classes/` + - Writes `META-INF/mal-expressions.txt` and `META-INF/mal-filter-expressions.properties` + +### Phase 3: LAL Transpiler + +**New module: `oap-server/analyzer/mal-lal-v2/lal-transpiler/`** + +1. **`LalToJavaTranspiler.java`** (~950 lines) + - Same AST approach as MAL but statement-based emission + - Core methods: + - `transpile(String className, String dslText)` -> generates Java source for `LalExpression` + - `emitStatement(Statement node, String receiver, BindingContext ctx)` -> statement emitter + - `visitConditionExpr(Expression node)` -> boolean expression emitter + - `emitPropertyAccess(PropertyExpression node)` -> `getAt()` with null safety + - SHA-256 deduplication: identical DSL content shares one class + - Helper methods in generated class: `getAt()`, `toLong()`, `toInt()`, `toBoolean()`, `isTruthy()`, `isNonEmptyString()` + +2. **Maven integration**: same `exec-maven-plugin` approach + - Writes `META-INF/lal-expressions.txt` (SHA-256 hash -> FQCN) + +### Phase 4: Runtime Loading (v2 Modules) + +**New module: `oap-server/analyzer/mal-lal-v2/meter-analyzer-v2/`** + +1. **Modified `DSL.java`** (MAL runtime): + ```java + public static Expression parse(String metricName, String expression) { + Map manifest = loadManifest("META-INF/mal-expressions.txt"); + String className = manifest.get(metricName); + MalExpression malExpr = (MalExpression) Class.forName(className) + .getDeclaredConstructor().newInstance(); + return new Expression(metricName, expression, malExpr); + } + ``` + +2. **Modified `Expression.java`** (MAL runtime): + - Wraps `MalExpression` instead of `DelegatingScript` + - `run()` calls `malExpression.run(sampleFamilies)` directly + - No `ExpandoMetaClass`, no `ExpressionDelegate`, no `ThreadLocal` + +3. **Modified `FilterExpression.java`** (MAL runtime): + - Loads `MalFilter` from `META-INF/mal-filter-expressions.properties` + - `filter()` calls `malFilter::test` via `SampleFamily.filter(SampleFilter)` + +**New module: `oap-server/analyzer/mal-lal-v2/log-analyzer-v2/`** + +4. **Modified `DSL.java`** (LAL runtime): + - Computes SHA-256 of DSL text, loads class from `META-INF/lal-expressions.txt` + - `evaluate()` calls `lalExpression.execute(filterSpec, binding)` directly + - No `GroovyShell`, no `LALDelegatingScript` + +### Phase 5: Hierarchy v1/v2 Module Split + +**Step 5a: Refactor `server-core` to remove Groovy** + +**File to modify:** `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java` + +1. Change `MatchingRule.closure` from `Closure` to `BiFunction matcher` +2. Add constructor that accepts the `BiFunction` matcher directly +3. Replace `getClosure()` with `match(Service upper, Service lower)` method +4. Change `init()` to accept a rule registry (`Map>`) from outside instead of calling `GroovyShell.evaluate()` internally +5. Remove all `groovy.lang.*` imports + +**File to modify:** `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java` + +6. Change `matchingRule.getClosure().call(service, comparedService)` (lines 201-203, 220-222) to `matchingRule.match(service, comparedService)` + +**Step 5b: Create hierarchy-v1 module** + +**New module:** `oap-server/analyzer/hierarchy-v1/` + +1. `GroovyHierarchyRuleProvider.java`: wraps original `GroovyShell.evaluate()` logic +2. Takes `Map` (rule name -> Groovy expression) from YAML +3. Returns `Map>` by evaluating closures +4. Depends on Groovy -- NOT included in runtime, only used by checker + +**Step 5c: Create hierarchy-v2 module** + +**New module:** `oap-server/analyzer/hierarchy-v2/` + +1. `JavaHierarchyRuleProvider.java`: static `RULE_REGISTRY` with 4 Java lambdas +2. Takes `Map` (rule name -> Groovy expression) from YAML (expression ignored, only name used for registry lookup) +3. Returns `Map>` from the registry +4. Fails fast with `IllegalArgumentException` for unknown rule names +5. Zero Groovy dependency + +### Phase 6: Comparison Test Suites + +**New module: `oap-server/analyzer/mal-lal-v1-v2-checker/`** + +1. **MAL comparison tests** (73 test classes): + - Base class `MALScriptComparisonBase` runs dual-path comparison: + - Path A: Fresh Groovy compilation with upstream `CompilerConfiguration` + - Path B: Load transpiled `MalExpression` from manifest + - Both receive identical `Map` input + - Compare: `ExpressionParsingContext` (scope, function, datatype, downsampling), `SampleFamily` result (values, labels, entity descriptions) + - JUnit 5 `@TestFactory` generates `DynamicTest` per metric rule + - Test data must be non-trivial to prevent vacuous agreement (both returning empty/null) + +2. **LAL comparison tests** (5 test classes): + - Base class `LALScriptComparisonBase` runs dual-path comparison: + - Path A: Groovy with `@CompileStatic` + `LALPrecompiledExtension` + - Path B: Load `LalExpression` from manifest via SHA-256 + - Compare: `shouldAbort()`, `shouldSave()`, LogData.Builder state, metrics container, `databaseSlowStatement`, `sampledTraceBuilder` + +3. **Test statistics**: 1,281 MAL assertions + 19 LAL assertions = 1,300 total + +**New module: `oap-server/analyzer/hierarchy-v1-v2-checker/`** + +4. **Hierarchy comparison tests**: + - Load rule expressions from `hierarchy-definition.yml` + - For each of the 4 rules: + - Path A: `GroovyHierarchyRuleProvider.buildRules()` (v1, Groovy closures) + - Path B: `JavaHierarchyRuleProvider.buildRules()` (v2, Java lambdas) + - Construct test `Service` pairs covering matching and non-matching cases: + - `name`: exact match `("svc", "svc")` -> true, `("svc", "other")` -> false + - `short-name`: shortName match/mismatch + - `lower-short-name-remove-ns`: `"svc"` vs `"svc.namespace"` -> true, no dot -> false, empty -> false + - `lower-short-name-with-fqdn`: `"db:3306"` vs `"db.svc.cluster.local"` -> true, no colon -> false, wrong suffix -> false + - Assert `v1.match(u, l) == v2.match(u, l)` for all test pairs + +### Phase 7: Cleanup and Dependency Removal + +1. **Move v1 code to `mal-lal-v1/` and `hierarchy-v1/`** (or mark as `test`) +2. **Remove Groovy from runtime classpath**: `groovy-5.0.3.jar` (~7 MB) becomes test-only +3. **Remove from `server-starter` dependencies**: replace v1 with v2 module references for MAL, LAL, and hierarchy +4. **Remove `NumberClosure.java`**: no longer needed without `ExpandoMetaClass` +5. **Remove `ExpressionDelegate.propertyMissing()`**: replaced by `samples.getOrDefault()` +6. **Remove Groovy closure overloads from `SampleFamily`** (after v1 is fully deprecated) +7. **Remove `LALDelegatingScript.java`**: replaced by `LalExpression` interface +8. **Verify `server-core` has zero Groovy imports**: `HierarchyDefinitionService` and `HierarchyService` now use `BiFunction` only + +--- + +## 6. What Gets Removed from Runtime + +| Component | Current | After | +|-----------|---------|-------| +| `GroovyShell.parse()` in MAL `DSL.java` | 1,250+ calls at boot | `Class.forName()` from manifest | +| `GroovyShell.evaluate()` in MAL `FilterExpression.java` | 29 filter compilations | `Class.forName()` from manifest | +| `GroovyShell.parse()` in LAL `DSL.java` | 10 script compilations | `Class.forName()` from manifest | +| `GroovyShell.evaluate()` in `HierarchyDefinitionService` | 4 rule compilations | `hierarchy-v2` Java lambda registry | +| `Closure` in `MatchingRule` | Groovy closure in `server-core` | `BiFunction` (Groovy-free `server-core`) | +| `ExpandoMetaClass` registration in `Expression.empower()` | Runtime metaclass on `Number` | Direct `multiply()`/`div()` method calls | +| `ExpressionDelegate.propertyMissing()` | Dynamic property dispatch | `samples.getOrDefault()` | +| `groovy.lang.Closure` in `SampleFamily` | 5 method signatures | Java functional interfaces | +| `groovy-5.0.3.jar` runtime dependency | ~7 MB on classpath | Removed (build-time only) | + +--- + +## 7. Transpiler Technical Details + +### 7.1 AST Parsing Strategy + +Both transpilers use Groovy's `CompilationUnit` at `Phases.CONVERSION`: + +```java +CompilationUnit cu = new CompilationUnit(); +cu.addSource("expression", new StringReaderSource( + new StringReader(groovyCode), cu.getConfiguration())); +cu.compile(Phases.CONVERSION); // Parse + AST transform, no codegen +ModuleNode ast = cu.getAST(); +``` + +This extracts the complete syntax tree without: +- Generating Groovy bytecode +- Resolving classes on classpath +- Activating MOP or MetaClass + +The Groovy dependency is therefore **build-time only**. + +### 7.2 MAL Arithmetic Operand Swap + +The transpiler must replicate the exact behavior of upstream's `ExpandoMetaClass` on `Number`. When a `Number` appears on the left side of an operator with a `SampleFamily` on the right, the operands must be handled carefully: + +``` +N + SF -> SF.plus(N) // commutative, swap operands +N - SF -> SF.minus(N).negative() // non-commutative: (N - SF) = -(SF - N) +N * SF -> SF.multiply(N) // commutative, swap operands +N / SF -> SF.newValue(v -> N / v) // non-commutative: per-sample (N / sample_value) +``` + +The transpiler detects `Number` vs `SampleFamily` operand types by tracking whether a sub-expression references sample names (metric properties) or is a numeric literal/constant. + +### 7.3 Batch Compilation + +Generated Java sources are compiled in a single `javac` invocation via `javax.tools.JavaCompiler`: + +```java +JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); +List sources = /* all generated .java files */; +compiler.getTask(null, fileManager, diagnostics, options, null, sources).call(); +``` + +This avoids 1,250+ individual `javac` invocations and provides full cross-file type checking. + +### 7.4 Manifest Format + +**`META-INF/mal-expressions.txt`**: +``` +metric_name_1=org.apache.skywalking.oap.meter.analyzer.dsl.generated.MalExpr_metric_name_1 +metric_name_2=org.apache.skywalking.oap.meter.analyzer.dsl.generated.MalExpr_metric_name_2 +... +``` + +**`META-INF/mal-filter-expressions.properties`**: +``` +{ tags -> tags.job_name == 'mysql-monitoring' }=org.apache.skywalking.oap.meter.analyzer.dsl.generated.MalFilter_0a1b2c3d +... +``` + +**`META-INF/lal-expressions.txt`**: +``` +sha256_hash_1=org.apache.skywalking.oap.log.analyzer.dsl.generated.LalExpr_sha256_hash_1 +sha256_hash_2=org.apache.skywalking.oap.log.analyzer.dsl.generated.LalExpr_sha256_hash_2 +... +``` + +--- + +## 8. Worked Examples + +### 8.1 MAL Expression: K8s Node CPU Capacity + +**Groovy (upstream):** +```groovy +kube_node_status_capacity.tagEqual('resource', 'cpu') + .sum(['node']) + .tag({tags -> tags.node_name = tags.node; tags.remove("node")}) + .service(['node_name'], Layer.K8S) +``` + +**Transpiled Java:** +```java +public class MalExpr_k8s_node_cpu implements MalExpression { + @Override + public SampleFamily run(Map samples) { + return samples.getOrDefault("kube_node_status_capacity", SampleFamily.EMPTY) + .tagEqual("resource", "cpu") + .sum(List.of("node")) + .tag(tags -> { + tags.put("node_name", tags.get("node")); + tags.remove("node"); + return tags; + }) + .service(List.of("node_name"), Layer.K8S); + } +} +``` + +### 8.2 MAL Expression: Arithmetic with Number on Left + +**Groovy (upstream):** +```groovy +100 - container_cpu_usage / container_resource_limit_cpu * 100 +``` + +**Transpiled Java:** +```java +public class MalExpr_cpu_percent implements MalExpression { + @Override + public SampleFamily run(Map samples) { + return samples.getOrDefault("container_cpu_usage", SampleFamily.EMPTY) + .div(samples.getOrDefault("container_resource_limit_cpu", SampleFamily.EMPTY)) + .multiply(100) + .minus(100) + .negative(); + } +} +``` + +### 8.3 MAL Filter Expression + +**Groovy (upstream):** +```groovy +{ tags -> tags.job_name == 'mysql-monitoring' } +``` + +**Transpiled Java:** +```java +public class MalFilter_mysql implements MalFilter { + @Override + public boolean test(Map tags) { + return "mysql-monitoring".equals(tags.get("job_name")); + } +} +``` + +### 8.4 LAL Expression: MySQL Slow SQL + +**Groovy (upstream):** +```groovy +filter { + json {} + extractor { + layer parsed.layer as String + service parsed.service as String + timestamp parsed.time as String + if (tag("LOG_KIND") == "SLOW_SQL") { + slowSql { + id parsed.id as String + statement parsed.statement as String + latency parsed.query_time as Long + } + } + } + sink {} +} +``` + +**Transpiled Java:** +```java +public class LalExpr_mysql_slowsql implements LalExpression { + + @Override + public void execute(FilterSpec filterSpec, Binding binding) { + filterSpec.json(); + filterSpec.extractor(ext -> { + ext.layer(String.valueOf(getAt(binding.parsed(), "layer"))); + ext.service(String.valueOf(getAt(binding.parsed(), "service"))); + ext.timestamp(String.valueOf(getAt(binding.parsed(), "time"))); + if ("SLOW_SQL".equals(ext.tag("LOG_KIND"))) { + ext.slowSql(ss -> { + ss.id(String.valueOf(getAt(binding.parsed(), "id"))); + ss.statement(String.valueOf(getAt(binding.parsed(), "statement"))); + ss.latency(toLong(getAt(binding.parsed(), "query_time"))); + }); + } + }); + filterSpec.sink(s -> {}); + } + + private static Object getAt(Object obj, String key) { + if (obj instanceof Binding.Parsed) return ((Binding.Parsed) obj).getAt(key); + if (obj instanceof Map) return ((Map) obj).get(key); + return null; + } + + private static long toLong(Object val) { + if (val instanceof Number) return ((Number) val).longValue(); + if (val instanceof String) return Long.parseLong((String) val); + return 0L; + } +} +``` + +### 8.5 Hierarchy Rule: lower-short-name-remove-ns + +**Groovy (upstream, in hierarchy-definition.yml):** +```groovy +{ (u, l) -> { + if(l.shortName.lastIndexOf('.') > 0) + return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); + return false; +} } +``` + +**Java replacement (in HierarchyDefinitionService.java):** +```java +RULE_REGISTRY.put("lower-short-name-remove-ns", (u, l) -> { + String sn = l.getShortName(); + int dot = sn.lastIndexOf('.'); + return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); +}); +``` + +--- + +## 9. Verification Strategy + +### 9.1 Dual-Path Comparison Testing + +Every generated Java class is validated against the original Groovy behavior in CI: + +``` +For each MAL YAML file: + For each metric rule: + 1. Compile expression with Groovy (v1 path) + 2. Load transpiled MalExpression (v2 path) + 3. Construct realistic sample data (non-trivial to prevent vacuous agreement) + 4. Run both paths with identical input + 5. Assert identical output: + - ExpressionParsingContext (scope, function, datatype, samples, downsampling) + - SampleFamily result (values, labels, entity descriptions) +``` + +### 9.2 Staleness Detection + +Properties files record SHA-256 hashes of upstream classes that have same-FQCN replacements. If upstream changes a class, the staleness test fails, forcing review of the replacement. + +### 9.3 Automatic Coverage + +New MAL/LAL YAML rules added to `server-starter/src/main/resources/` are automatically covered by the transpiler and comparison tests -- if the transpiler produces different results from Groovy, the build fails. + +--- + +## 10. Statistics + +| Metric | Count | +|--------|-------| +| MAL YAML files processed | 71 | +| MAL metric expressions transpiled | 1,254 | +| MAL filter expressions transpiled | 29 | +| LAL YAML files processed | 8 | +| LAL rules transpiled | 10 (6 unique after SHA-256 dedup) | +| Hierarchy rules replaced | 4 | +| Hierarchy rules replaced | 4 | +| Total generated Java classes | ~1,289 | +| Comparison test assertions | 1,300+ (MAL: 1,281, LAL: 19, hierarchy: 4 rules x multiple service pairs) | +| Lines of transpiler code (MAL) | ~1,230 | +| Lines of transpiler code (LAL) | ~950 | +| Runtime JAR removed | groovy-5.0.3.jar (~7 MB) | + +--- + +## 11. Risk Assessment + +| Risk | Mitigation | +|------|-----------| +| Transpiler misses an AST pattern | 1,300 dual-path comparison tests catch any divergence | +| New MAL/LAL expression uses unsupported Groovy syntax | Transpiler throws clear error at build time; new pattern must be added | +| Upstream SampleFamily/Spec changes break replacement | Staleness tests detect SHA-256 changes | +| Performance regression | Eliminated dynamic dispatch should only improve performance; benchmark with `MetricConvert` pipeline | +| Custom user MAL/LAL scripts | Users who extend default rules with custom Groovy scripts must follow the same syntax subset supported by the transpiler | + +--- + +## 12. Migration Timeline + +1. **Phase 1**: Add interfaces and functional interface overloads to existing `meter-analyzer` and `log-analyzer` (non-breaking, additive changes) +2. **Phase 2-3**: Implement MAL and LAL transpilers in new `mal-lal-v2/` modules +3. **Phase 4**: Implement v2 runtime loaders (modified `DSL.java`, `Expression.java`, `FilterExpression.java`) +4. **Phase 5**: Hierarchy v1/v2 module split -- refactor `server-core` to remove Groovy, create `hierarchy-v1/` (Groovy, checker-only) and `hierarchy-v2/` (Java lambdas, runtime) +5. **Phase 6**: Build comparison test suites -- `mal-lal-v1-v2-checker/` AND `hierarchy-v1-v2-checker/` +6. **Phase 7**: Switch `server-starter` from v1 to v2 for all three subsystems (MAL, LAL, hierarchy), remove Groovy from runtime classpath +7. **Eventually**: Remove `mal-lal-v1`, `hierarchy-v1`, and all checker modules once community confidence is established From 2f928fcf50c8b1a2f0c20ccfa114914a93d71a64 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 14:22:14 +0800 Subject: [PATCH 02/64] Add pure Java interfaces and functional overloads for MAL/LAL (Phase 1) Add MalExpression, MalFilter, LalExpression functional interfaces and SampleFamilyFunctions (TagFunction, SampleFilter, ForEachFunction, DecorateFunction, PropertiesExtractor). Add Java functional interface overloads alongside existing Groovy Closure methods in SampleFamily, FilterSpec, ExtractorSpec, and SinkSpec. Change InstanceEntityDescription to use Function instead of Closure. All 129 existing tests pass. Co-Authored-By: Claude Opus 4.6 --- .../oap/log/analyzer/dsl/LalExpression.java | 30 ++++ .../dsl/spec/extractor/ExtractorSpec.java | 88 ++++++++++ .../analyzer/dsl/spec/filter/FilterSpec.java | 166 ++++++++++++++++++ .../log/analyzer/dsl/spec/sink/SinkSpec.java | 22 +++ .../InstanceEntityDescription.java | 4 +- .../oap/meter/analyzer/dsl/MalExpression.java | 30 ++++ .../oap/meter/analyzer/dsl/MalFilter.java | 30 ++++ .../oap/meter/analyzer/dsl/SampleFamily.java | 93 +++++++++- .../analyzer/dsl/SampleFamilyFunctions.java | 77 ++++++++ 9 files changed, 536 insertions(+), 4 deletions(-) create mode 100644 oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java new file mode 100644 index 000000000000..f96b02f485a2 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl; + +import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; + +/** + * Pure Java replacement for Groovy-based LAL DelegatingScript. + * Each transpiled LAL expression implements this interface. + */ +@FunctionalInterface +public interface LalExpression { + void execute(FilterSpec filterSpec, Binding binding); +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java index ee9e58e72e74..2a51d10d4f1b 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java @@ -29,6 +29,7 @@ import java.util.Map; import java.util.Objects; import java.util.Optional; +import java.util.function.Consumer; import java.util.stream.Collectors; import lombok.experimental.Delegate; import org.apache.commons.lang3.StringUtils; @@ -333,6 +334,93 @@ public void sampledTrace(@DelegatesTo(SampledTraceSpec.class) final Closure c sourceReceiver.receive(entity); } + public void metrics(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + final SampleBuilder builder = new SampleBuilder(); + consumer.accept(builder); + + final Sample sample = builder.build(); + final SampleFamily sampleFamily = SampleFamilyBuilder.newBuilder(sample).build(); + + final Optional> possibleMetricsContainer = BINDING.get().metricsContainer(); + + if (possibleMetricsContainer.isPresent()) { + possibleMetricsContainer.get().add(sampleFamily); + } else { + metricConverts.forEach(it -> it.toMeter( + ImmutableMap.builder() + .put(sample.getName(), sampleFamily) + .build() + )); + } + } + + public void slowSql(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + LogData.Builder log = BINDING.get().log(); + if (log.getLayer() == null + || log.getService() == null + || log.getTimestamp() < 1) { + LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); + return; + } + DatabaseSlowStatementBuilder builder = new DatabaseSlowStatementBuilder(namingControl); + builder.setLayer(Layer.nameOf(log.getLayer())); + + builder.setServiceName(log.getService()); + + BINDING.get().databaseSlowStatement(builder); + + consumer.accept(slowSql); + + if (builder.getId() == null + || builder.getLatency() < 1 + || builder.getStatement() == null) { + LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); + return; + } + + long timeBucketForDB = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Second); + builder.setTimeBucket(timeBucketForDB); + builder.setTimestamp(log.getTimestamp()); + + builder.prepare(); + sourceReceiver.receive(builder.toDatabaseSlowStatement()); + + ServiceMeta serviceMeta = new ServiceMeta(); + serviceMeta.setName(builder.getServiceName()); + serviceMeta.setLayer(builder.getLayer()); + long timeBucket = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Minute); + serviceMeta.setTimeBucket(timeBucket); + sourceReceiver.receive(serviceMeta); + } + + public void sampledTrace(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + LogData.Builder log = BINDING.get().log(); + SampledTraceBuilder builder = new SampledTraceBuilder(namingControl); + builder.setLayer(log.getLayer()); + builder.setTimestamp(log.getTimestamp()); + builder.setServiceName(log.getService()); + builder.setServiceInstanceName(log.getServiceInstance()); + builder.setTraceId(log.getTraceContext().getTraceId()); + BINDING.get().sampledTrace(builder); + + consumer.accept(sampledTrace); + + builder.validate(); + final Record record = builder.toRecord(); + final ISource entity = builder.toEntity(); + RecordStreamProcessor.getInstance().in(record); + sourceReceiver.receive(entity); + } + public static class SampleBuilder { @Delegate private final Sample.SampleBuilder sampleBuilder = Sample.builder(); diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java index 7fb7557b7558..b83e71b8b849 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java @@ -28,6 +28,7 @@ import java.util.Map; import java.util.Optional; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; import org.apache.skywalking.apm.network.logging.v3.LogData; import org.apache.skywalking.oap.log.analyzer.dsl.Binding; @@ -189,4 +190,169 @@ public void sink(@DelegatesTo(SinkSpec.class) final Closure cl) { public void filter(final Closure cl) { cl.call(); } + + public void text(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(textParser); + } + + public void text() { + if (BINDING.get().shouldAbort()) { + return; + } + } + + public void json(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(jsonParser); + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = jsonParser.create().readValue( + logData.getBody().getJson().getJson(), parsedType + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (jsonParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void json() { + if (BINDING.get().shouldAbort()) { + return; + } + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = jsonParser.create().readValue( + logData.getBody().getJson().getJson(), parsedType + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (jsonParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void yaml(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(yamlParser); + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = yamlParser.create().load( + logData.getBody().getYaml().getYaml() + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (yamlParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void yaml() { + if (BINDING.get().shouldAbort()) { + return; + } + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = yamlParser.create().load( + logData.getBody().getYaml().getYaml() + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (yamlParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void extractor(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(extractor); + } + + public void sink(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(sink); + + final Binding b = BINDING.get(); + final LogData.Builder logData = b.log(); + final Message extraLog = b.extraLog(); + + if (!b.shouldSave()) { + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Log is dropped: {}", TextFormat.shortDebugString(logData)); + } + return; + } + + final Optional> container = BINDING.get().logContainer(); + if (container.isPresent()) { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .filter(it -> it instanceof RecordSinkListener) + .map(it -> it.parse(logData, extraLog)) + .map(it -> (RecordSinkListener) it) + .map(RecordSinkListener::getLog) + .findFirst() + .ifPresent(log -> container.get().set(log)); + } else { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .forEach(it -> it.parse(logData, extraLog).build()); + } + } + + public void sink() { + if (BINDING.get().shouldAbort()) { + return; + } + + final Binding b = BINDING.get(); + final LogData.Builder logData = b.log(); + final Message extraLog = b.extraLog(); + + if (!b.shouldSave()) { + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Log is dropped: {}", TextFormat.shortDebugString(logData)); + } + return; + } + + final Optional> container = BINDING.get().logContainer(); + if (container.isPresent()) { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .filter(it -> it instanceof RecordSinkListener) + .map(it -> it.parse(logData, extraLog)) + .map(it -> (RecordSinkListener) it) + .map(RecordSinkListener::getLog) + .findFirst() + .ifPresent(log -> container.get().set(log)); + } else { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .forEach(it -> it.parse(logData, extraLog).build()); + } + } + + public void abort() { + BINDING.get().abort(); + } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java index 82566f9b25d1..f2ae371a21e2 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java @@ -20,6 +20,7 @@ import groovy.lang.Closure; import groovy.lang.DelegatesTo; +import java.util.function.Consumer; import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; import org.apache.skywalking.oap.server.library.module.ModuleManager; @@ -44,6 +45,13 @@ public void sampler(@DelegatesTo(SamplerSpec.class) final Closure cl) { cl.call(); } + public void sampler(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(sampler); + } + @SuppressWarnings("unused") public void enforcer(final Closure cl) { if (BINDING.get().shouldAbort()) { @@ -52,6 +60,13 @@ public void enforcer(final Closure cl) { BINDING.get().save(); } + public void enforcer() { + if (BINDING.get().shouldAbort()) { + return; + } + BINDING.get().save(); + } + @SuppressWarnings("unused") public void dropper(final Closure cl) { if (BINDING.get().shouldAbort()) { @@ -59,4 +74,11 @@ public void dropper(final Closure cl) { } BINDING.get().drop(); } + + public void dropper() { + if (BINDING.get().shouldAbort()) { + return; + } + BINDING.get().drop(); + } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java index 04c0a2a5dad6..83f1e5f87f23 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java @@ -20,6 +20,7 @@ import java.util.List; import java.util.Map; +import java.util.function.Function; import java.util.stream.Collectors; import java.util.stream.Stream; import lombok.Getter; @@ -27,7 +28,6 @@ import lombok.ToString; import org.apache.skywalking.oap.server.core.analysis.Layer; import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; -import groovy.lang.Closure; @Getter @RequiredArgsConstructor @@ -39,7 +39,7 @@ public class InstanceEntityDescription implements EntityDescription { private final Layer layer; private final String serviceDelimiter; private final String instanceDelimiter; - private final Closure> propertiesExtractor; + private final Function, Map> propertiesExtractor; @Override public List getLabelKeys() { diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java new file mode 100644 index 000000000000..cfaf1d5beee5 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.Map; + +/** + * Pure Java replacement for Groovy-based MAL DelegatingScript. + * Each transpiled MAL expression implements this interface. + */ +@FunctionalInterface +public interface MalExpression { + SampleFamily run(Map samples); +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java new file mode 100644 index 000000000000..9d8eaa259e92 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.Map; + +/** + * Pure Java replacement for Groovy Closure-based MAL filter expressions. + * Each transpiled filter expression implements this interface. + */ +@FunctionalInterface +public interface MalFilter { + boolean test(Map tags); +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java index f72fed4cef8d..5a979d0a381d 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java @@ -60,6 +60,11 @@ import groovy.lang.Closure; import io.vavr.Function2; import io.vavr.Function3; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.DecorateFunction; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.ForEachFunction; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.PropertiesExtractor; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.SampleFilter; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.TagFunction; import lombok.AccessLevel; import lombok.Builder; import lombok.EqualsAndHashCode; @@ -400,6 +405,26 @@ public SampleFamily tag(Closure cl) { ); } + @SuppressWarnings(value = "unchecked") + public SampleFamily tag(TagFunction fn) { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .map(sample -> { + Map arg = Maps.newHashMap(sample.labels); + Map r = fn.apply(arg); + return sample.toBuilder() + .labels( + ImmutableMap.copyOf( + Optional.ofNullable(r).orElse(arg))) + .build(); + }).toArray(Sample[]::new) + ); + } + public SampleFamily filter(Closure filter) { if (this == EMPTY) { return EMPTY; @@ -413,6 +438,19 @@ public SampleFamily filter(Closure filter) { return SampleFamily.build(context, filtered); } + public SampleFamily filter(SampleFilter filter) { + if (this == EMPTY) { + return EMPTY; + } + final Sample[] filtered = Arrays.stream(samples) + .filter(it -> filter.test(it.labels)) + .toArray(Sample[]::new); + if (filtered.length == 0) { + return EMPTY; + } + return SampleFamily.build(context, filtered); + } + /* k8s retags*/ public SampleFamily retagByK8sMeta(String newLabelName, K8sRetagType type, @@ -516,12 +554,30 @@ public SampleFamily instance(List serviceKeys, String serviceDelimiter, if (this == EMPTY) { return EMPTY; } + return createMeterSamples(new InstanceEntityDescription( + serviceKeys, instanceKeys, layer, serviceDelimiter, instanceDelimiter, + propertiesExtractor == null ? null : propertiesExtractor::call)); + } + + public SampleFamily instance(List serviceKeys, String serviceDelimiter, + List instanceKeys, String instanceDelimiter, + Layer layer, PropertiesExtractor propertiesExtractor) { + Preconditions.checkArgument(serviceKeys.size() > 0); + Preconditions.checkArgument(instanceKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE_INSTANCE; + ctx.scopeLabels.addAll(serviceKeys); + ctx.scopeLabels.addAll(instanceKeys); + }); + if (this == EMPTY) { + return EMPTY; + } return createMeterSamples(new InstanceEntityDescription( serviceKeys, instanceKeys, layer, serviceDelimiter, instanceDelimiter, propertiesExtractor)); } public SampleFamily instance(List serviceKeys, List instanceKeys, Layer layer) { - return instance(serviceKeys, Const.POINT, instanceKeys, Const.POINT, layer, null); + return instance(serviceKeys, Const.POINT, instanceKeys, Const.POINT, layer, (Closure>) null); } public SampleFamily endpoint(List serviceKeys, List endpointKeys, String delimiter, Layer layer) { @@ -601,6 +657,19 @@ public SampleFamily forEach(List array, Closure each) { }).toArray(Sample[]::new)); } + public SampleFamily forEach(List array, ForEachFunction each) { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build(this.context, Arrays.stream(this.samples).map(sample -> { + Map labels = Maps.newHashMap(sample.getLabels()); + for (String element : array) { + each.accept(element, labels); + } + return sample.toBuilder().labels(ImmutableMap.copyOf(labels)).build(); + }).toArray(Sample[]::new)); + } + public SampleFamily processRelation(String detectPointKey, List serviceKeys, List instanceKeys, String sourceProcessIdKey, String destProcessIdKey, String componentKey) { Preconditions.checkArgument(serviceKeys.size() > 0); Preconditions.checkArgument(instanceKeys.size() > 0); @@ -717,6 +786,26 @@ public SampleFamily decorate(Closure c) { return this; } + public SampleFamily decorate(DecorateFunction c) { + ExpressionParsingContext.get().ifPresent(ctx -> { + if (ctx.getScopeType() != ScopeType.SERVICE) { + throw new IllegalStateException("decorate() should be invoked after service()"); + } + if (ctx.isHistogram()) { + throw new IllegalStateException("decorate() not supported for histogram metrics"); + } + }); + if (this == EMPTY) { + return EMPTY; + } + this.context.getMeterSamples().keySet().forEach(meterEntity -> { + if (meterEntity.getScopeType().equals(ScopeType.SERVICE)) { + c.accept(meterEntity); + } + }); + return this; + } + /** * The parsing context holds key results more than sample collection. */ @@ -777,7 +866,7 @@ private static MeterEntity buildMeterEntity(List samples, InstanceEntityDescription instanceEntityDescription = (InstanceEntityDescription) entityDescription; Map properties = null; if (instanceEntityDescription.getPropertiesExtractor() != null) { - properties = instanceEntityDescription.getPropertiesExtractor().call(samples.get(0).labels); + properties = instanceEntityDescription.getPropertiesExtractor().apply(samples.get(0).labels); } return MeterEntity.newServiceInstance( InternalOps.dim(samples, instanceEntityDescription.getServiceKeys(), instanceEntityDescription.getServiceDelimiter()), diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java new file mode 100644 index 000000000000..1c9747b56a25 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.Map; +import java.util.function.Consumer; +import java.util.function.Function; +import java.util.function.Predicate; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity; + +/** + * Pure Java functional interfaces replacing Groovy Closure parameters in SampleFamily methods. + */ +public final class SampleFamilyFunctions { + + private SampleFamilyFunctions() { + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#tag(groovy.lang.Closure)}. + * Receives a mutable label map and returns the (possibly modified) map. + */ + @FunctionalInterface + public interface TagFunction extends Function, Map> { + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#filter(groovy.lang.Closure)}. + * Tests whether a sample's labels match the filter criteria. + */ + @FunctionalInterface + public interface SampleFilter extends Predicate> { + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#forEach(java.util.List, groovy.lang.Closure)}. + * Called for each element in the array with the element value and a mutable labels map. + */ + @FunctionalInterface + public interface ForEachFunction { + void accept(String element, Map tags); + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#decorate(groovy.lang.Closure)}. + * Decorates service meter entities. + */ + @FunctionalInterface + public interface DecorateFunction extends Consumer { + } + + /** + * Replaces {@code Closure>} in + * {@link SampleFamily#instance(java.util.List, String, java.util.List, String, + * org.apache.skywalking.oap.server.core.analysis.Layer, groovy.lang.Closure)}. + * Extracts instance properties from sample labels. + */ + @FunctionalInterface + public interface PropertiesExtractor extends Function, Map> { + } +} From 0170c109bf64997eb662336f734795bd81613588 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 14:47:20 +0800 Subject: [PATCH 03/64] Add MAL transpiler module for build-time Groovy-to-Java conversion (Phase 2) Ports MalToJavaTranspiler from skywalking-graalvm-distro into a new mal-transpiler analyzer submodule. The transpiler parses Groovy MAL expressions/filters via AST at CONVERSION phase and emits equivalent Java classes implementing MalExpression/MalFilter interfaces from Phase 1. Co-Authored-By: Claude Opus 4.6 --- oap-server/analyzer/mal-transpiler/pom.xml | 41 + .../transpiler/mal/MalToJavaTranspiler.java | 1099 +++++++++++++++++ .../mal/MalToJavaTranspilerTest.java | 904 ++++++++++++++ oap-server/analyzer/pom.xml | 1 + 4 files changed, 2045 insertions(+) create mode 100644 oap-server/analyzer/mal-transpiler/pom.xml create mode 100644 oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java create mode 100644 oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java diff --git a/oap-server/analyzer/mal-transpiler/pom.xml b/oap-server/analyzer/mal-transpiler/pom.xml new file mode 100644 index 000000000000..4f45b67acb77 --- /dev/null +++ b/oap-server/analyzer/mal-transpiler/pom.xml @@ -0,0 +1,41 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + mal-transpiler + + + + org.apache.skywalking + meter-analyzer + ${project.version} + + + org.apache.groovy + groovy + + + diff --git a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java new file mode 100644 index 000000000000..c256bb589cb6 --- /dev/null +++ b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java @@ -0,0 +1,1099 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.transpiler.mal; + +import java.io.File; +import java.io.IOException; +import java.io.StringWriter; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; +import javax.tools.JavaCompiler; +import javax.tools.JavaFileObject; +import javax.tools.StandardJavaFileManager; +import javax.tools.ToolProvider; +import lombok.extern.slf4j.Slf4j; +import org.codehaus.groovy.ast.ClassNode; +import org.codehaus.groovy.ast.ModuleNode; +import org.codehaus.groovy.ast.Parameter; +import org.codehaus.groovy.ast.expr.ArgumentListExpression; +import org.codehaus.groovy.ast.expr.BinaryExpression; +import org.codehaus.groovy.ast.expr.ClosureExpression; +import org.codehaus.groovy.ast.expr.ClassExpression; +import org.codehaus.groovy.ast.expr.ConstantExpression; +import org.codehaus.groovy.ast.expr.DeclarationExpression; +import org.codehaus.groovy.ast.expr.ElvisOperatorExpression; +import org.codehaus.groovy.ast.expr.Expression; +import org.codehaus.groovy.ast.expr.ListExpression; +import org.codehaus.groovy.ast.expr.MapEntryExpression; +import org.codehaus.groovy.ast.expr.MapExpression; +import org.codehaus.groovy.ast.expr.MethodCallExpression; +import org.codehaus.groovy.ast.expr.PropertyExpression; +import org.codehaus.groovy.ast.expr.TernaryExpression; +import org.codehaus.groovy.ast.expr.TupleExpression; +import org.codehaus.groovy.ast.expr.VariableExpression; +import org.codehaus.groovy.syntax.Types; +import org.codehaus.groovy.ast.expr.BooleanExpression; +import org.codehaus.groovy.ast.expr.NotExpression; +import org.codehaus.groovy.ast.stmt.BlockStatement; +import org.codehaus.groovy.ast.stmt.EmptyStatement; +import org.codehaus.groovy.ast.stmt.ExpressionStatement; +import org.codehaus.groovy.ast.stmt.IfStatement; +import org.codehaus.groovy.ast.stmt.ReturnStatement; +import org.codehaus.groovy.ast.stmt.Statement; +import org.codehaus.groovy.control.CompilationUnit; +import org.codehaus.groovy.control.CompilerConfiguration; +import org.codehaus.groovy.control.Phases; + +/** + * Transpiles Groovy MAL expressions to Java source code at build time. + * Parses expression strings into Groovy AST (via CompilationUnit at CONVERSION phase), + * walks AST nodes, and produces equivalent Java classes implementing MalExpression or MalFilter. + * + *

Supported AST patterns: + *

    + *
  • Variable references: sample family lookups, DownsamplingType constants, KNOWN_TYPES
    • + *
  • Method chains: .sum(), .service(), .tagEqual(), .rate(), .histogram(), etc.
    • + *
  • Binary arithmetic with operand-swap logic per upstream ExpandoMetaClass + * (N-SF: sf.minus(N).negative(), N/SF: sf.newValue(v->N/v))
    • + *
  • tag() closures: TagFunction lambda (assignment, remove, string concat, if/else)
    • + *
  • Filter closures: MalFilter class (==, !=, in, truthiness, negation, &&, ||)
    • + *
  • forEach() closures: ForEachFunction lambda (var decls, if/else-if, early return)
    • + *
  • instance() with PropertiesExtractor closure: Map.of() from map literals
    • + *
  • Elvis (?:), safe navigation (?.), ternary (? :)
    • + *
  • Batch compilation via javax.tools.JavaCompiler + manifest writing
    • + *
+ */ +@Slf4j +public class MalToJavaTranspiler { + + static final String GENERATED_PACKAGE = + "org.apache.skywalking.oap.server.core.source.oal.rt.mal"; + + private static final Set DOWNSAMPLING_CONSTANTS = Set.of( + "AVG", "SUM", "LATEST", "SUM_PER_MIN", "MAX", "MIN" + ); + + private static final Set KNOWN_TYPES = Set.of( + "Layer", "DetectPoint", "K8sRetagType", "ProcessRegistry", "TimeUnit" + ); + + // ---- Batch state tracking ---- + + private final Map expressionSources = new LinkedHashMap<>(); + + private final Map filterSources = new LinkedHashMap<>(); + + private final Map filterLiteralToClass = new LinkedHashMap<>(); + + /** + * Transpile a MAL expression to a Java class source implementing MalExpression. + * + * @param className simple class name (e.g. "MalExpr_meter_jvm_heap") + * @param expression the Groovy expression string + * @return generated Java source code + */ + public String transpileExpression(final String className, final String expression) { + final ModuleNode ast = parseToAST(expression); + final Statement body = extractBody(ast); + + final Set sampleNames = new LinkedHashSet<>(); + collectSampleNames(body, sampleNames); + + final String javaBody = visitStatement(body); + + final StringBuilder sb = new StringBuilder(); + sb.append("package ").append(GENERATED_PACKAGE).append(";\n\n"); + sb.append("import java.util.*;\n"); + sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.*;\n"); + sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.*;\n"); + sb.append("import org.apache.skywalking.oap.server.core.analysis.Layer;\n"); + sb.append("import org.apache.skywalking.oap.server.core.source.DetectPoint;\n"); + sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType;\n"); + sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry;\n\n"); + + sb.append("public class ").append(className).append(" implements MalExpression {\n"); + sb.append(" @Override\n"); + sb.append(" public SampleFamily run(Map samples) {\n"); + + if (!sampleNames.isEmpty()) { + sb.append(" ExpressionParsingContext.get().ifPresent(ctx -> {\n"); + for (String name : sampleNames) { + sb.append(" ctx.getSamples().add(\"").append(escapeJava(name)).append("\");\n"); + } + sb.append(" });\n"); + } + + sb.append(" return ").append(javaBody).append(";\n"); + sb.append(" }\n"); + sb.append("}\n"); + + return sb.toString(); + } + + /** + * Transpile a MAL filter literal to a Java class source implementing MalFilter. + * Filter literals are closures like: { tags -> tags.job_name == 'vm-monitoring' } + * + * @param className simple class name (e.g. "MalFilter_0") + * @param filterLiteral the Groovy closure literal string + * @return generated Java source code + */ + public String transpileFilter(final String className, final String filterLiteral) { + final ModuleNode ast = parseToAST(filterLiteral); + final Statement body = extractBody(ast); + + // The filter literal is a closure expression at the top level + final ClosureExpression closure = extractClosure(body); + final Parameter[] params = closure.getParameters(); + final String tagsVar = (params != null && params.length > 0) ? params[0].getName() : "tags"; + + // Get the body expression — may need to unwrap inner block/closure + final Expression bodyExpr = extractFilterBodyExpr(closure.getCode(), tagsVar); + + // Generate the boolean condition + final String condition = visitFilterCondition(bodyExpr, tagsVar); + + final StringBuilder sb = new StringBuilder(); + sb.append("package ").append(GENERATED_PACKAGE).append(";\n\n"); + sb.append("import java.util.*;\n"); + sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.*;\n\n"); + + sb.append("public class ").append(className).append(" implements MalFilter {\n"); + sb.append(" @Override\n"); + sb.append(" public boolean test(Map tags) {\n"); + sb.append(" return ").append(condition).append(";\n"); + sb.append(" }\n"); + sb.append("}\n"); + + return sb.toString(); + } + + private ClosureExpression extractClosure(final Statement body) { + final List stmts = getStatements(body); + if (stmts.size() == 1 && stmts.get(0) instanceof ExpressionStatement) { + final Expression expr = ((ExpressionStatement) stmts.get(0)).getExpression(); + if (expr instanceof ClosureExpression) { + return (ClosureExpression) expr; + } + } + throw new IllegalStateException( + "Filter literal must be a single closure expression, got: " + + (stmts.isEmpty() ? "empty" : stmts.get(0).getClass().getSimpleName())); + } + + private Expression extractFilterBodyExpr(final Statement code, final String tagsVar) { + final List stmts = getStatements(code); + if (stmts.isEmpty()) { + throw new IllegalStateException("Empty filter closure body"); + } + + final Statement last = stmts.get(stmts.size() - 1); + Expression expr; + if (last instanceof ExpressionStatement) { + expr = ((ExpressionStatement) last).getExpression(); + } else if (last instanceof ReturnStatement) { + expr = ((ReturnStatement) last).getExpression(); + } else if (last instanceof BlockStatement) { + return extractFilterBodyExpr(last, tagsVar); + } else { + throw new UnsupportedOperationException( + "Unsupported filter body statement: " + last.getClass().getSimpleName()); + } + + if (expr instanceof ClosureExpression) { + final ClosureExpression inner = (ClosureExpression) expr; + return extractFilterBodyExpr(inner.getCode(), tagsVar); + } + + return expr; + } + + // ---- AST Parsing ---- + + ModuleNode parseToAST(final String expression) { + final CompilerConfiguration cc = new CompilerConfiguration(); + final CompilationUnit cu = new CompilationUnit(cc); + cu.addSource("Script", expression); + cu.compile(Phases.CONVERSION); + final List modules = cu.getAST().getModules(); + if (modules.isEmpty()) { + throw new IllegalStateException("No AST modules produced for: " + expression); + } + return modules.get(0); + } + + Statement extractBody(final ModuleNode module) { + final BlockStatement block = module.getStatementBlock(); + if (block != null && !block.getStatements().isEmpty()) { + return block; + } + final List classes = module.getClasses(); + if (!classes.isEmpty()) { + return module.getStatementBlock(); + } + throw new IllegalStateException("Empty AST body"); + } + + // ---- Sample Name Collection ---- + + private void collectSampleNames(final Statement stmt, final Set names) { + if (stmt instanceof BlockStatement) { + for (Statement s : ((BlockStatement) stmt).getStatements()) { + collectSampleNames(s, names); + } + } else if (stmt instanceof ExpressionStatement) { + collectSampleNamesFromExpr(((ExpressionStatement) stmt).getExpression(), names); + } else if (stmt instanceof ReturnStatement) { + collectSampleNamesFromExpr(((ReturnStatement) stmt).getExpression(), names); + } + } + + void collectSampleNamesFromExpr(final Expression expr, final Set names) { + if (expr instanceof VariableExpression) { + final String name = ((VariableExpression) expr).getName(); + if (!DOWNSAMPLING_CONSTANTS.contains(name) + && !KNOWN_TYPES.contains(name) + && !name.equals("this") && !name.equals("time")) { + names.add(name); + } + } else if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + collectSampleNamesFromExpr(bin.getLeftExpression(), names); + collectSampleNamesFromExpr(bin.getRightExpression(), names); + } else if (expr instanceof PropertyExpression) { + final PropertyExpression pe = (PropertyExpression) expr; + collectSampleNamesFromExpr(pe.getObjectExpression(), names); + } else if (expr instanceof MethodCallExpression) { + final MethodCallExpression mce = (MethodCallExpression) expr; + collectSampleNamesFromExpr(mce.getObjectExpression(), names); + collectSampleNamesFromExpr(mce.getArguments(), names); + } else if (expr instanceof ArgumentListExpression) { + for (Expression e : ((ArgumentListExpression) expr).getExpressions()) { + collectSampleNamesFromExpr(e, names); + } + } else if (expr instanceof TupleExpression) { + for (Expression e : ((TupleExpression) expr).getExpressions()) { + collectSampleNamesFromExpr(e, names); + } + } + } + + // ---- Statement Visiting ---- + + String visitStatement(final Statement stmt) { + if (stmt instanceof BlockStatement) { + final List stmts = ((BlockStatement) stmt).getStatements(); + if (stmts.size() == 1) { + return visitStatement(stmts.get(0)); + } + // Multi-statement: last one is the return value + return visitStatement(stmts.get(stmts.size() - 1)); + } else if (stmt instanceof ExpressionStatement) { + return visitExpression(((ExpressionStatement) stmt).getExpression()); + } else if (stmt instanceof ReturnStatement) { + return visitExpression(((ReturnStatement) stmt).getExpression()); + } + throw new UnsupportedOperationException( + "Unsupported statement: " + stmt.getClass().getSimpleName()); + } + + // ---- Expression Visiting ---- + + String visitExpression(final Expression expr) { + if (expr instanceof VariableExpression) { + return visitVariable((VariableExpression) expr); + } else if (expr instanceof ConstantExpression) { + return visitConstant((ConstantExpression) expr); + } else if (expr instanceof MethodCallExpression) { + return visitMethodCall((MethodCallExpression) expr); + } else if (expr instanceof PropertyExpression) { + return visitProperty((PropertyExpression) expr); + } else if (expr instanceof ListExpression) { + return visitList((ListExpression) expr); + } else if (expr instanceof BinaryExpression) { + return visitBinary((BinaryExpression) expr); + } else if (expr instanceof ClosureExpression) { + throw new UnsupportedOperationException( + "Bare ClosureExpression outside method call context: " + expr.getText()); + } + throw new UnsupportedOperationException( + "Unsupported expression (not yet implemented): " + + expr.getClass().getSimpleName() + " = " + expr.getText()); + } + + private String visitVariable(final VariableExpression expr) { + final String name = expr.getName(); + if (DOWNSAMPLING_CONSTANTS.contains(name)) { + return "DownsamplingType." + name; + } + if (KNOWN_TYPES.contains(name)) { + return name; + } + if (name.equals("this")) { + return "this"; + } + // Sample family lookup + return "samples.getOrDefault(\"" + escapeJava(name) + "\", SampleFamily.EMPTY)"; + } + + private String visitConstant(final ConstantExpression expr) { + final Object value = expr.getValue(); + if (value == null) { + return "null"; + } + if (value instanceof String) { + return "\"" + escapeJava((String) value) + "\""; + } + if (value instanceof Integer) { + return value.toString(); + } + if (value instanceof Long) { + return value + "L"; + } + if (value instanceof Double) { + return value.toString(); + } + if (value instanceof Float) { + return value + "f"; + } + if (value instanceof Boolean) { + return value.toString(); + } + return value.toString(); + } + + // ---- MethodCall, Property, List ---- + + private String visitMethodCall(final MethodCallExpression expr) { + final String methodName = expr.getMethodAsString(); + final Expression objExpr = expr.getObjectExpression(); + final ArgumentListExpression args = toArgList(expr.getArguments()); + + // tag(closure) -> TagFunction lambda + if ("tag".equals(methodName) && args.getExpressions().size() == 1 + && args.getExpression(0) instanceof ClosureExpression) { + final String obj = visitExpression(objExpr); + final String lambda = visitTagClosure((ClosureExpression) args.getExpression(0)); + return obj + ".tag((TagFunction) " + lambda + ")"; + } + + // forEach(list, closure) -> ForEachFunction lambda + if ("forEach".equals(methodName) && args.getExpressions().size() == 2 + && args.getExpression(1) instanceof ClosureExpression) { + final String obj = visitExpression(objExpr); + final String list = visitExpression(args.getExpression(0)); + final String lambda = visitForEachClosure((ClosureExpression) args.getExpression(1)); + return obj + ".forEach(" + list + ", (ForEachFunction) " + lambda + ")"; + } + + // instance(..., closure) -> last arg is PropertiesExtractor lambda + if ("instance".equals(methodName) && !args.getExpressions().isEmpty()) { + final Expression lastArg = args.getExpression(args.getExpressions().size() - 1); + if (lastArg instanceof ClosureExpression) { + final String obj = visitExpression(objExpr); + final List argStrs = new ArrayList<>(); + for (int i = 0; i < args.getExpressions().size() - 1; i++) { + argStrs.add(visitExpression(args.getExpression(i))); + } + final String lambda = visitPropertiesExtractorClosure((ClosureExpression) lastArg); + argStrs.add("(PropertiesExtractor) " + lambda); + return obj + ".instance(" + String.join(", ", argStrs) + ")"; + } + } + + final String obj = visitExpression(objExpr); + + // Static method calls: ClassExpression.method(...) + if (objExpr instanceof ClassExpression) { + final String typeName = objExpr.getType().getNameWithoutPackage(); + final List argStrs = visitArgList(args); + return typeName + "." + methodName + "(" + String.join(", ", argStrs) + ")"; + } + + // Regular instance method call: obj.method(args) + final List argStrs = visitArgList(args); + return obj + "." + methodName + "(" + String.join(", ", argStrs) + ")"; + } + + private String visitProperty(final PropertyExpression expr) { + final Expression obj = expr.getObjectExpression(); + final String prop = expr.getPropertyAsString(); + + if (obj instanceof ClassExpression) { + return obj.getType().getNameWithoutPackage() + "." + prop; + } + if (obj instanceof VariableExpression) { + final String varName = ((VariableExpression) obj).getName(); + if (KNOWN_TYPES.contains(varName)) { + return varName + "." + prop; + } + } + + return visitExpression(obj) + "." + prop; + } + + private String visitList(final ListExpression expr) { + final List elements = new ArrayList<>(); + for (Expression e : expr.getExpressions()) { + elements.add(visitExpression(e)); + } + return "List.of(" + String.join(", ", elements) + ")"; + } + + // ---- Binary Arithmetic ---- + + private String visitBinary(final BinaryExpression expr) { + final int opType = expr.getOperation().getType(); + + if (isArithmetic(opType)) { + return visitArithmetic(expr.getLeftExpression(), expr.getRightExpression(), opType); + } + + throw new UnsupportedOperationException( + "Unsupported binary operator (not yet implemented): " + + expr.getOperation().getText() + " in " + expr.getText()); + } + + /** + * Arithmetic with operand-swap logic per upstream ExpandoMetaClass: + * 
+     *   SF + SF  -> left.plus(right)
+     *   SF - SF  -> left.minus(right)
+     *   SF * SF  -> left.multiply(right)
+     *   SF / SF  -> left.div(right)
+     *   SF op N  -> sf.op(N)
+     *   N + SF   -> sf.plus(N)          (swap)
+     *   N - SF   -> sf.minus(N).negative()
+     *   N * SF   -> sf.multiply(N)      (swap)
+     *   N / SF   -> sf.newValue(v -> N / v)
+     *   N op N   -> plain arithmetic
+     *
+ */ + private String visitArithmetic(final Expression left, final Expression right, final int opType) { + final boolean leftNum = isNumberLiteral(left); + final boolean rightNum = isNumberLiteral(right); + final String leftStr = visitExpression(left); + final String rightStr = visitExpression(right); + + if (leftNum && rightNum) { + return "(" + leftStr + " " + opSymbol(opType) + " " + rightStr + ")"; + } + + if (!leftNum && rightNum) { + return leftStr + "." + opMethod(opType) + "(" + rightStr + ")"; + } + + if (leftNum && !rightNum) { + switch (opType) { + case Types.PLUS: + return rightStr + ".plus(" + leftStr + ")"; + case Types.MINUS: + return rightStr + ".minus(" + leftStr + ").negative()"; + case Types.MULTIPLY: + return rightStr + ".multiply(" + leftStr + ")"; + case Types.DIVIDE: + return rightStr + ".newValue(v -> " + leftStr + " / v)"; + default: + break; + } + } + + // SF op SF + return leftStr + "." + opMethod(opType) + "(" + rightStr + ")"; + } + + private boolean isNumberLiteral(final Expression expr) { + if (expr instanceof ConstantExpression) { + return ((ConstantExpression) expr).getValue() instanceof Number; + } + return false; + } + + private boolean isArithmetic(final int opType) { + return opType == Types.PLUS || opType == Types.MINUS + || opType == Types.MULTIPLY || opType == Types.DIVIDE; + } + + private String opMethod(final int opType) { + switch (opType) { + case Types.PLUS: return "plus"; + case Types.MINUS: return "minus"; + case Types.MULTIPLY: return "multiply"; + case Types.DIVIDE: return "div"; + default: return "???"; + } + } + + private String opSymbol(final int opType) { + switch (opType) { + case Types.PLUS: return "+"; + case Types.MINUS: return "-"; + case Types.MULTIPLY: return "*"; + case Types.DIVIDE: return "/"; + default: return "?"; + } + } + + // ---- tag() Closure ---- + + private String visitTagClosure(final ClosureExpression closure) { + final Parameter[] params = closure.getParameters(); + final String tagsVar = (params != null && params.length > 0) ? params[0].getName() : "tags"; + final List stmts = getStatements(closure.getCode()); + + final StringBuilder sb = new StringBuilder(); + sb.append("(").append(tagsVar).append(" -> {\n"); + for (Statement s : stmts) { + sb.append(" ").append(visitTagStatement(s, tagsVar)).append("\n"); + } + sb.append(" return ").append(tagsVar).append(";\n"); + sb.append(" })"); + return sb.toString(); + } + + private String visitTagStatement(final Statement stmt, final String tagsVar) { + if (stmt instanceof ExpressionStatement) { + return visitTagExpr(((ExpressionStatement) stmt).getExpression(), tagsVar) + ";"; + } + if (stmt instanceof ReturnStatement) { + return "return " + tagsVar + ";"; + } + if (stmt instanceof IfStatement) { + return visitTagIf((IfStatement) stmt, tagsVar); + } + throw new UnsupportedOperationException( + "Unsupported tag closure statement: " + stmt.getClass().getSimpleName()); + } + + // ---- If/Else + Compound Conditions in tag() ---- + + private String visitTagIf(final IfStatement ifStmt, final String tagsVar) { + final String condition = visitTagCondition(ifStmt.getBooleanExpression().getExpression(), tagsVar); + final List ifBody = getStatements(ifStmt.getIfBlock()); + final Statement elseBlock = ifStmt.getElseBlock(); + + final StringBuilder sb = new StringBuilder(); + sb.append("if (").append(condition).append(") {\n"); + for (Statement s : ifBody) { + sb.append(" ").append(visitTagStatement(s, tagsVar)).append("\n"); + } + sb.append(" }"); + + if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { + sb.append(" else {\n"); + final List elseBody = getStatements(elseBlock); + for (Statement s : elseBody) { + sb.append(" ").append(visitTagStatement(s, tagsVar)).append("\n"); + } + sb.append(" }"); + } + + return sb.toString(); + } + + private String visitTagCondition(final Expression expr, final String tagsVar) { + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + final int opType = bin.getOperation().getType(); + + if (opType == Types.COMPARE_EQUAL) { + return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, false); + } + if (opType == Types.COMPARE_NOT_EQUAL) { + return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, true); + } + if (opType == Types.LOGICAL_OR) { + return visitTagCondition(bin.getLeftExpression(), tagsVar) + + " || " + visitTagCondition(bin.getRightExpression(), tagsVar); + } + if (opType == Types.LOGICAL_AND) { + return visitTagCondition(bin.getLeftExpression(), tagsVar) + + " && " + visitTagCondition(bin.getRightExpression(), tagsVar); + } + } + if (expr instanceof BooleanExpression) { + return visitTagCondition(((BooleanExpression) expr).getExpression(), tagsVar); + } + return visitTagValue(expr, tagsVar); + } + + private String visitTagEquals(final Expression left, final Expression right, + final String tagsVar, final boolean negate) { + if (isNullConstant(right)) { + final String leftStr = visitTagValue(left, tagsVar); + return negate ? leftStr + " != null" : leftStr + " == null"; + } + if (isNullConstant(left)) { + final String rightStr = visitTagValue(right, tagsVar); + return negate ? rightStr + " != null" : rightStr + " == null"; + } + + final String leftStr = visitTagValue(left, tagsVar); + final String rightStr = visitTagValue(right, tagsVar); + + if (right instanceof ConstantExpression && ((ConstantExpression) right).getValue() instanceof String) { + final String result = rightStr + ".equals(" + leftStr + ")"; + return negate ? "!" + result : result; + } + if (left instanceof ConstantExpression && ((ConstantExpression) left).getValue() instanceof String) { + final String result = leftStr + ".equals(" + rightStr + ")"; + return negate ? "!" + result : result; + } + final String result = "Objects.equals(" + leftStr + ", " + rightStr + ")"; + return negate ? "!" + result : result; + } + + // ---- Filter Conditions ---- + + private String visitFilterCondition(final Expression expr, final String tagsVar) { + if (expr instanceof NotExpression) { + final Expression inner = ((NotExpression) expr).getExpression(); + final String val = visitTagValue(inner, tagsVar); + return "(" + val + " == null || " + val + ".isEmpty())"; + } + + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + final int opType = bin.getOperation().getType(); + + if (opType == Types.COMPARE_EQUAL) { + return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, false); + } + if (opType == Types.COMPARE_NOT_EQUAL) { + return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, true); + } + if (opType == Types.LOGICAL_OR) { + return visitFilterCondition(bin.getLeftExpression(), tagsVar) + + " || " + visitFilterCondition(bin.getRightExpression(), tagsVar); + } + if (opType == Types.LOGICAL_AND) { + return visitFilterCondition(bin.getLeftExpression(), tagsVar) + + " && " + visitFilterCondition(bin.getRightExpression(), tagsVar); + } + if (opType == Types.KEYWORD_IN) { + final String val = visitTagValue(bin.getLeftExpression(), tagsVar); + final String list = visitTagValue(bin.getRightExpression(), tagsVar); + return list + ".contains(" + val + ")"; + } + } + + if (expr instanceof BooleanExpression) { + return visitFilterCondition(((BooleanExpression) expr).getExpression(), tagsVar); + } + + final String val = visitTagValue(expr, tagsVar); + return "(" + val + " != null && !" + val + ".isEmpty())"; + } + + private String visitTagExpr(final Expression expr, final String tagsVar) { + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + if (bin.getOperation().getType() == Types.ASSIGN) { + return visitTagAssignment(bin.getLeftExpression(), bin.getRightExpression(), tagsVar); + } + } + if (expr instanceof MethodCallExpression) { + final MethodCallExpression mce = (MethodCallExpression) expr; + if ("remove".equals(mce.getMethodAsString()) && isTagsVar(mce.getObjectExpression(), tagsVar)) { + final ArgumentListExpression args = toArgList(mce.getArguments()); + return tagsVar + ".remove(" + visitTagValue(args.getExpression(0), tagsVar) + ")"; + } + } + return visitTagValue(expr, tagsVar); + } + + private String visitTagAssignment(final Expression left, final Expression right, final String tagsVar) { + final String val = visitTagValue(right, tagsVar); + + if (left instanceof PropertyExpression) { + final PropertyExpression prop = (PropertyExpression) left; + if (isTagsVar(prop.getObjectExpression(), tagsVar)) { + return tagsVar + ".put(\"" + escapeJava(prop.getPropertyAsString()) + "\", " + val + ")"; + } + } + if (left instanceof BinaryExpression) { + final BinaryExpression sub = (BinaryExpression) left; + if (sub.getOperation().getType() == Types.LEFT_SQUARE_BRACKET + && isTagsVar(sub.getLeftExpression(), tagsVar)) { + final String key = visitTagValue(sub.getRightExpression(), tagsVar); + return tagsVar + ".put(" + key + ", " + val + ")"; + } + } + throw new UnsupportedOperationException( + "Unsupported tag assignment target: " + left.getClass().getSimpleName() + " = " + left.getText()); + } + + String visitTagValue(final Expression expr, final String tagsVar) { + if (expr instanceof PropertyExpression) { + final PropertyExpression prop = (PropertyExpression) expr; + if (isTagsVar(prop.getObjectExpression(), tagsVar)) { + return tagsVar + ".get(\"" + escapeJava(prop.getPropertyAsString()) + "\")"; + } + return visitProperty(prop); + } + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + if (bin.getOperation().getType() == Types.LEFT_SQUARE_BRACKET + && isTagsVar(bin.getLeftExpression(), tagsVar)) { + return tagsVar + ".get(" + visitTagValue(bin.getRightExpression(), tagsVar) + ")"; + } + if (bin.getOperation().getType() == Types.PLUS) { + return visitTagValue(bin.getLeftExpression(), tagsVar) + + " + " + visitTagValue(bin.getRightExpression(), tagsVar); + } + } + // Elvis operator — must check BEFORE TernaryExpression since it extends it + if (expr instanceof ElvisOperatorExpression) { + final ElvisOperatorExpression elvis = (ElvisOperatorExpression) expr; + final String val = visitTagValue(elvis.getTrueExpression(), tagsVar); + final String defaultVal = visitTagValue(elvis.getFalseExpression(), tagsVar); + return "(" + val + " != null ? " + val + " : " + defaultVal + ")"; + } + if (expr instanceof TernaryExpression) { + final TernaryExpression tern = (TernaryExpression) expr; + final String cond = visitFilterCondition(tern.getBooleanExpression().getExpression(), tagsVar); + final String trueVal = visitTagValue(tern.getTrueExpression(), tagsVar); + final String falseVal = visitTagValue(tern.getFalseExpression(), tagsVar); + return "(" + cond + " ? " + trueVal + " : " + falseVal + ")"; + } + if (expr instanceof MethodCallExpression) { + final MethodCallExpression mce = (MethodCallExpression) expr; + final String obj = visitTagValue(mce.getObjectExpression(), tagsVar); + final ArgumentListExpression args = toArgList(mce.getArguments()); + final List argStrs = new ArrayList<>(); + for (Expression a : args.getExpressions()) { + argStrs.add(visitTagValue(a, tagsVar)); + } + final String call = obj + "." + mce.getMethodAsString() + "(" + String.join(", ", argStrs) + ")"; + if (mce.isSafe()) { + return "(" + obj + " != null ? " + call + " : null)"; + } + return call; + } + if (expr instanceof VariableExpression) { + final String name = ((VariableExpression) expr).getName(); + if (name.equals(tagsVar)) { + return tagsVar; + } + return name; + } + if (expr instanceof ConstantExpression) { + return visitConstant((ConstantExpression) expr); + } + if (expr instanceof ListExpression) { + return visitList((ListExpression) expr); + } + if (expr instanceof MapExpression) { + final MapExpression map = (MapExpression) expr; + final List entries = new ArrayList<>(); + for (MapEntryExpression entry : map.getMapEntryExpressions()) { + entries.add(visitTagValue(entry.getKeyExpression(), tagsVar)); + entries.add(visitTagValue(entry.getValueExpression(), tagsVar)); + } + return "Map.of(" + String.join(", ", entries) + ")"; + } + return visitExpression(expr); + } + + private boolean isTagsVar(final Expression expr, final String tagsVar) { + return expr instanceof VariableExpression + && ((VariableExpression) expr).getName().equals(tagsVar); + } + + private List getStatements(final Statement stmt) { + if (stmt instanceof BlockStatement) { + return ((BlockStatement) stmt).getStatements(); + } + return List.of(stmt); + } + + // ---- forEach() Closure ---- + + private String visitForEachClosure(final ClosureExpression closure) { + final Parameter[] params = closure.getParameters(); + final String prefixVar = (params != null && params.length > 0) ? params[0].getName() : "prefix"; + final String tagsVar = (params != null && params.length > 1) ? params[1].getName() : "tags"; + + final List stmts = getStatements(closure.getCode()); + + final StringBuilder sb = new StringBuilder(); + sb.append("(").append(prefixVar).append(", ").append(tagsVar).append(") -> {\n"); + for (Statement s : stmts) { + sb.append(" ").append(visitForEachStatement(s, tagsVar)).append("\n"); + } + sb.append(" }"); + return sb.toString(); + } + + private String visitForEachStatement(final Statement stmt, final String tagsVar) { + if (stmt instanceof ExpressionStatement) { + return visitForEachExpr(((ExpressionStatement) stmt).getExpression(), tagsVar) + ";"; + } + if (stmt instanceof ReturnStatement) { + return "return;"; + } + if (stmt instanceof IfStatement) { + return visitForEachIf((IfStatement) stmt, tagsVar); + } + throw new UnsupportedOperationException( + "Unsupported forEach closure statement: " + stmt.getClass().getSimpleName()); + } + + private String visitForEachExpr(final Expression expr, final String tagsVar) { + if (expr instanceof DeclarationExpression) { + final DeclarationExpression decl = (DeclarationExpression) expr; + final String typeName = decl.getVariableExpression().getType().getNameWithoutPackage(); + final String varName = decl.getVariableExpression().getName(); + final String init = visitTagValue(decl.getRightExpression(), tagsVar); + return typeName + " " + varName + " = " + init; + } + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + if (bin.getOperation().getType() == Types.ASSIGN) { + final Expression left = bin.getLeftExpression(); + if (isTagWrite(left, tagsVar)) { + return visitTagAssignment(left, bin.getRightExpression(), tagsVar); + } + if (left instanceof VariableExpression) { + return ((VariableExpression) left).getName() + + " = " + visitTagValue(bin.getRightExpression(), tagsVar); + } + } + } + return visitTagExpr(expr, tagsVar); + } + + private String visitForEachIf(final IfStatement ifStmt, final String tagsVar) { + final String condition = visitTagCondition(ifStmt.getBooleanExpression().getExpression(), tagsVar); + final List ifBody = getStatements(ifStmt.getIfBlock()); + final Statement elseBlock = ifStmt.getElseBlock(); + + final StringBuilder sb = new StringBuilder(); + sb.append("if (").append(condition).append(") {\n"); + for (Statement s : ifBody) { + sb.append(" ").append(visitForEachStatement(s, tagsVar)).append("\n"); + } + sb.append(" }"); + + if (elseBlock instanceof IfStatement) { + sb.append(" else ").append(visitForEachIf((IfStatement) elseBlock, tagsVar)); + } else if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { + sb.append(" else {\n"); + for (Statement s : getStatements(elseBlock)) { + sb.append(" ").append(visitForEachStatement(s, tagsVar)).append("\n"); + } + sb.append(" }"); + } + + return sb.toString(); + } + + private boolean isTagWrite(final Expression left, final String tagsVar) { + if (left instanceof PropertyExpression) { + return isTagsVar(((PropertyExpression) left).getObjectExpression(), tagsVar); + } + if (left instanceof BinaryExpression) { + final BinaryExpression sub = (BinaryExpression) left; + return sub.getOperation().getType() == Types.LEFT_SQUARE_BRACKET + && isTagsVar(sub.getLeftExpression(), tagsVar); + } + return false; + } + + private boolean isNullConstant(final Expression expr) { + return expr instanceof ConstantExpression && ((ConstantExpression) expr).getValue() == null; + } + + // ---- PropertiesExtractor Closure ---- + + private String visitPropertiesExtractorClosure(final ClosureExpression closure) { + final Parameter[] params = closure.getParameters(); + final String tagsVar = (params != null && params.length > 0) ? params[0].getName() : "tags"; + final List stmts = getStatements(closure.getCode()); + + final Statement last = stmts.get(stmts.size() - 1); + Expression bodyExpr; + if (last instanceof ExpressionStatement) { + bodyExpr = ((ExpressionStatement) last).getExpression(); + } else if (last instanceof ReturnStatement) { + bodyExpr = ((ReturnStatement) last).getExpression(); + } else { + throw new UnsupportedOperationException( + "Unsupported PropertiesExtractor closure body: " + last.getClass().getSimpleName()); + } + return "(" + tagsVar + " -> " + visitTagValue(bodyExpr, tagsVar) + ")"; + } + + // ---- Batch Registration, Compilation, and Manifest Writing ---- + + public void registerExpression(final String className, final String source) { + expressionSources.put(className, source); + } + + public void registerFilter(final String className, final String filterLiteral, final String source) { + filterSources.put(className, source); + filterLiteralToClass.put(filterLiteral, GENERATED_PACKAGE + "." + className); + } + + /** + * Compile all registered sources using javax.tools.JavaCompiler. + * + * @param sourceDir directory to write .java source files (package dirs created automatically) + * @param outputDir directory for compiled .class files + * @param classpath classpath for javac (semicolon/colon-separated JAR paths) + * @throws IOException if file I/O fails + */ + public void compileAll(final File sourceDir, final File outputDir, + final String classpath) throws IOException { + final Map allSources = new LinkedHashMap<>(); + allSources.putAll(expressionSources); + allSources.putAll(filterSources); + + if (allSources.isEmpty()) { + log.info("No MAL sources to compile."); + return; + } + + final String packageDir = GENERATED_PACKAGE.replace('.', File.separatorChar); + final File srcPkgDir = new File(sourceDir, packageDir); + if (!srcPkgDir.exists() && !srcPkgDir.mkdirs()) { + throw new IOException("Failed to create source dir: " + srcPkgDir); + } + if (!outputDir.exists() && !outputDir.mkdirs()) { + throw new IOException("Failed to create output dir: " + outputDir); + } + + final List javaFiles = new ArrayList<>(); + for (Map.Entry entry : allSources.entrySet()) { + final File javaFile = new File(srcPkgDir, entry.getKey() + ".java"); + Files.writeString(javaFile.toPath(), entry.getValue()); + javaFiles.add(javaFile); + } + + final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + if (compiler == null) { + throw new IllegalStateException("No Java compiler available — requires JDK"); + } + + final StringWriter errorWriter = new StringWriter(); + + try (StandardJavaFileManager fileManager = compiler.getStandardFileManager(null, null, null)) { + final Iterable compilationUnits = + fileManager.getJavaFileObjectsFromFiles(javaFiles); + + final List options = Arrays.asList( + "-d", outputDir.getAbsolutePath(), + "-classpath", classpath + ); + + final JavaCompiler.CompilationTask task = compiler.getTask( + errorWriter, fileManager, null, options, null, compilationUnits); + + final boolean success = task.call(); + if (!success) { + throw new RuntimeException( + "Java compilation failed for " + javaFiles.size() + " MAL sources:\n" + + errorWriter); + } + } + + log.info("Compiled {} MAL sources to {}", allSources.size(), outputDir); + } + + /** + * Write mal-expressions.txt manifest: one FQCN per line. + */ + public void writeExpressionManifest(final File outputDir) throws IOException { + final File manifestDir = new File(outputDir, "META-INF"); + if (!manifestDir.exists() && !manifestDir.mkdirs()) { + throw new IOException("Failed to create META-INF dir: " + manifestDir); + } + + final List lines = expressionSources.keySet().stream() + .map(name -> GENERATED_PACKAGE + "." + name) + .collect(Collectors.toList()); + Files.write(new File(manifestDir, "mal-expressions.txt").toPath(), lines); + log.info("Wrote mal-expressions.txt with {} entries", lines.size()); + } + + /** + * Write mal-filter-expressions.properties manifest: literal=FQCN. + */ + public void writeFilterManifest(final File outputDir) throws IOException { + final File manifestDir = new File(outputDir, "META-INF"); + if (!manifestDir.exists() && !manifestDir.mkdirs()) { + throw new IOException("Failed to create META-INF dir: " + manifestDir); + } + + final List lines = filterLiteralToClass.entrySet().stream() + .map(e -> escapeProperties(e.getKey()) + "=" + e.getValue()) + .collect(Collectors.toList()); + Files.write(new File(manifestDir, "mal-filter-expressions.properties").toPath(), lines); + log.info("Wrote mal-filter-expressions.properties with {} entries", lines.size()); + } + + private static String escapeProperties(final String s) { + return s.replace("\\", "\\\\") + .replace("=", "\\=") + .replace(":", "\\:") + .replace(" ", "\\ "); + } + + // ---- Argument Utilities ---- + + private ArgumentListExpression toArgList(final Expression args) { + if (args instanceof ArgumentListExpression) { + return (ArgumentListExpression) args; + } + if (args instanceof TupleExpression) { + final ArgumentListExpression ale = new ArgumentListExpression(); + for (Expression e : ((TupleExpression) args).getExpressions()) { + ale.addExpression(e); + } + return ale; + } + final ArgumentListExpression ale = new ArgumentListExpression(); + ale.addExpression(args); + return ale; + } + + private List visitArgList(final ArgumentListExpression args) { + final List result = new ArrayList<>(); + for (Expression arg : args.getExpressions()) { + result.add(visitExpression(arg)); + } + return result; + } + + // ---- Utility ---- + + static String escapeJava(final String s) { + return s.replace("\\", "\\\\") + .replace("\"", "\\\"") + .replace("\n", "\\n") + .replace("\r", "\\r") + .replace("\t", "\\t"); + } +} diff --git a/oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java b/oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java new file mode 100644 index 000000000000..0207d8df32d2 --- /dev/null +++ b/oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java @@ -0,0 +1,904 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.transpiler.mal; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MalToJavaTranspilerTest { + + private MalToJavaTranspiler transpiler; + + @BeforeEach + void setUp() { + transpiler = new MalToJavaTranspiler(); + } + + // ---- AST Parsing + Simple Variable References ---- + + @Test + void simpleVariableReference() { + final String java = transpiler.transpileExpression("MalExpr_test", "metric_name"); + assertNotNull(java); + + assertTrue(java.contains("package " + MalToJavaTranspiler.GENERATED_PACKAGE), + "Should have correct package"); + assertTrue(java.contains("public class MalExpr_test implements MalExpression"), + "Should implement MalExpression"); + assertTrue(java.contains("public SampleFamily run(Map samples)"), + "Should have run method"); + + assertTrue(java.contains("ctx.getSamples().add(\"metric_name\")"), + "Should track sample name in parsing context"); + + assertTrue(java.contains("samples.getOrDefault(\"metric_name\", SampleFamily.EMPTY)"), + "Should look up sample family from map"); + } + + @Test + void downsamplingConstantNotTrackedAsSample() { + final String java = transpiler.transpileExpression("MalExpr_test", "SUM"); + assertNotNull(java); + + assertTrue(!java.contains("ctx.getSamples().add(\"SUM\")"), + "Should not track DownsamplingType constant as sample"); + + assertTrue(java.contains("DownsamplingType.SUM"), + "Should resolve to DownsamplingType.SUM"); + } + + @Test + void parseToAST_producesModuleNode() { + final var ast = transpiler.parseToAST("some_metric"); + assertNotNull(ast, "Should produce a ModuleNode"); + assertNotNull(ast.getStatementBlock(), "Should have a statement block"); + } + + @Test + void constantString() { + final String java = transpiler.transpileExpression("MalExpr_test", "'hello'"); + assertNotNull(java); + assertTrue(java.contains("\"hello\""), + "Should convert Groovy string to Java string"); + } + + @Test + void constantNumber() { + final String java = transpiler.transpileExpression("MalExpr_test", "42"); + assertNotNull(java); + assertTrue(java.contains("42"), + "Should preserve number literal"); + } + + // ---- Method Chains + List Literals + Enum Properties ---- + + @Test + void simpleMethodChain() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric_name.sum(['a', 'b']).service(['svc'], Layer.GENERAL)"); + assertNotNull(java); + + assertTrue(java.contains(".sum(List.of(\"a\", \"b\"))"), + "Should translate ['a','b'] to List.of(\"a\", \"b\")"); + assertTrue(java.contains(".service(List.of(\"svc\"), Layer.GENERAL)"), + "Should translate Layer.GENERAL as enum"); + } + + @Test + void tagEqualChain() { + final String java = transpiler.transpileExpression("MalExpr_test", + "cpu_seconds.tagNotEqual('mode', 'idle').sum(['host']).rate('PT1M')"); + assertNotNull(java); + + assertTrue(java.contains(".tagNotEqual(\"mode\", \"idle\")"), + "Should translate tagNotEqual with string args"); + assertTrue(java.contains(".sum(List.of(\"host\"))"), + "Should translate single-element list"); + assertTrue(java.contains(".rate(\"PT1M\")"), + "Should translate rate with string arg"); + } + + @Test + void downsamplingMethod() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.downsampling(SUM)"); + assertNotNull(java); + + assertTrue(java.contains(".downsampling(DownsamplingType.SUM)"), + "Should resolve SUM to DownsamplingType.SUM"); + } + + @Test + void retagByK8sMeta() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.retagByK8sMeta('service', K8sRetagType.Pod2Service, 'pod', 'namespace')"); + assertNotNull(java); + + assertTrue(java.contains(".retagByK8sMeta(\"service\", K8sRetagType.Pod2Service, \"pod\", \"namespace\")"), + "Should translate K8sRetagType enum and string args"); + } + + @Test + void histogramPercentile() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.sum(['le', 'svc']).histogram().histogram_percentile([50, 75, 90])"); + assertNotNull(java); + + assertTrue(java.contains(".histogram()"), + "Should translate no-arg histogram()"); + assertTrue(java.contains(".histogram_percentile(List.of(50, 75, 90))"), + "Should translate integer list"); + } + + @Test + void sampleNameCollectionThroughChain() { + final String java = transpiler.transpileExpression("MalExpr_test", + "my_metric.sum(['a']).service(['svc'], Layer.GENERAL)"); + assertNotNull(java); + + assertTrue(java.contains("ctx.getSamples().add(\"my_metric\")"), + "Should collect sample name from root of method chain"); + assertTrue(!java.contains("ctx.getSamples().add(\"a\")"), + "Should NOT collect 'a' (it's a string constant arg, not a sample)"); + } + + @Test + void detectPointEnum() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.serviceRelation(DetectPoint.CLIENT, ['src'], ['dst'], Layer.MESH_DP)"); + assertNotNull(java); + + assertTrue(java.contains("DetectPoint.CLIENT"), + "Should translate DetectPoint enum"); + assertTrue(java.contains("Layer.MESH_DP"), + "Should translate Layer enum"); + } + + @Test + void enumImportsPresent() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.service(['svc'], Layer.GENERAL)"); + assertNotNull(java); + + assertTrue(java.contains("import org.apache.skywalking.oap.server.core.analysis.Layer;"), + "Should import Layer"); + assertTrue(java.contains("import org.apache.skywalking.oap.server.core.source.DetectPoint;"), + "Should import DetectPoint"); + assertTrue(java.contains("import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType;"), + "Should import K8sRetagType"); + } + + // ---- Binary Arithmetic with Operand-Swap ---- + + @Test + void sfTimesNumber() { + final String java = transpiler.transpileExpression("MalExpr_test", "metric * 100"); + assertNotNull(java); + assertTrue(java.contains(".multiply(100)"), + "SF * N should call .multiply(N)"); + } + + @Test + void sfDivNumber() { + final String java = transpiler.transpileExpression("MalExpr_test", "metric / 1024"); + assertNotNull(java); + assertTrue(java.contains(".div(1024)"), + "SF / N should call .div(N)"); + } + + @Test + void numberMinusSf() { + final String java = transpiler.transpileExpression("MalExpr_test", "100 - metric"); + assertNotNull(java); + assertTrue(java.contains(".minus(100).negative()"), + "N - SF should produce sf.minus(N).negative()"); + } + + @Test + void numberDivSf() { + final String java = transpiler.transpileExpression("MalExpr_test", "1 / metric"); + assertNotNull(java); + assertTrue(java.contains(".newValue(v -> 1 / v)"), + "N / SF should produce sf.newValue(v -> N / v)"); + } + + @Test + void numberPlusSf() { + final String java = transpiler.transpileExpression("MalExpr_test", "10 + metric"); + assertNotNull(java); + assertTrue(java.contains(".plus(10)"), + "N + SF should swap to sf.plus(N)"); + } + + @Test + void numberTimesSf() { + final String java = transpiler.transpileExpression("MalExpr_test", "100 * metric"); + assertNotNull(java); + assertTrue(java.contains(".multiply(100)"), + "N * SF should swap to sf.multiply(N)"); + } + + @Test + void sfMinusSf() { + final String java = transpiler.transpileExpression("MalExpr_test", "mem_total - mem_avail"); + assertNotNull(java); + assertTrue(java.contains("ctx.getSamples().add(\"mem_total\")"), + "Should collect both sample names"); + assertTrue(java.contains("ctx.getSamples().add(\"mem_avail\")"), + "Should collect both sample names"); + assertTrue(java.contains(".minus("), + "SF - SF should call .minus()"); + } + + @Test + void sfDivSfTimesNumber() { + final String java = transpiler.transpileExpression("MalExpr_test", + "used_bytes / max_bytes * 100"); + assertNotNull(java); + assertTrue(java.contains(".div("), + "Should have .div() for SF / SF"); + assertTrue(java.contains(".multiply(100)"), + "Should have .multiply(100) for result * 100"); + } + + @Test + void nestedParenArithmetic() { + final String java = transpiler.transpileExpression("MalExpr_test", + "100 - ((mem_free * 100) / mem_total)"); + assertNotNull(java); + assertTrue(java.contains(".multiply(100)"), + "Should have inner multiply"); + assertTrue(java.contains(".negative()"), + "100 - SF should produce .negative()"); + } + + @Test + void parenthesizedWithMethodChain() { + final String java = transpiler.transpileExpression("MalExpr_test", + "(metric * 100).tagNotEqual('mode', 'idle').sum(['host']).rate('PT1M')"); + assertNotNull(java); + assertTrue(java.contains(".multiply(100)"), + "Should have multiply inside parens"); + assertTrue(java.contains(".tagNotEqual(\"mode\", \"idle\")"), + "Should chain tagNotEqual after parens"); + assertTrue(java.contains(".rate(\"PT1M\")"), + "Should chain rate at the end"); + } + + // ---- tag() Closure — Simple Cases ---- + + @Test + void tagAssignmentWithStringConcat() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.route = 'route/' + tags['route']})"); + assertNotNull(java); + + assertTrue(java.contains(".tag((TagFunction)"), + "Should cast closure to TagFunction"); + assertTrue(java.contains("tags.put(\"route\", \"route/\" + tags.get(\"route\"))"), + "Should translate assignment with string concat and subscript read"); + assertTrue(java.contains("return tags;"), + "Should return tags at end of lambda"); + } + + @Test + void tagRemove() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.remove('condition')})"); + assertNotNull(java); + + assertTrue(java.contains("tags.remove(\"condition\")"), + "Should translate remove call"); + } + + @Test + void tagPropertyToProperty() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.rs_nm = tags.set})"); + assertNotNull(java); + + assertTrue(java.contains("tags.put(\"rs_nm\", tags.get(\"set\"))"), + "Should translate property read on RHS to tags.get()"); + } + + @Test + void tagStringConcatWithPropertyRead() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.cluster = 'es::' + tags.cluster})"); + assertNotNull(java); + + assertTrue(java.contains("tags.put(\"cluster\", \"es::\" + tags.get(\"cluster\"))"), + "Should translate string concat with property read"); + } + + @Test + void tagClosureLambdaStructure() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.x = 'y'})"); + assertNotNull(java); + + assertTrue(java.contains("(tags -> {"), + "Should have lambda opening"); + assertTrue(java.contains("return tags;"), + "Should return tags variable"); + } + + @Test + void tagWithSubscriptWrite() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags['service_name'] = tags['svc']})"); + assertNotNull(java); + + assertTrue(java.contains("tags.put(\"service_name\", tags.get(\"svc\"))"), + "Should translate subscript write and read"); + } + + @Test + void tagChainAfterTag() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.x = 'y'}).sum(['host']).service(['svc'], Layer.GENERAL)"); + assertNotNull(java); + + assertTrue(java.contains(".tag((TagFunction)"), + "Should have tag call"); + assertTrue(java.contains(".sum(List.of(\"host\"))"), + "Should chain sum after tag"); + assertTrue(java.contains(".service(List.of(\"svc\"), Layer.GENERAL)"), + "Should chain service after sum"); + } + + // ---- tag() Closure — if/else + Compound Conditions ---- + + @Test + void ifOnlyWithChainedOr() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['gc'] == 'PS Scavenge' || tags['gc'] == 'Copy' || tags['gc'] == 'ParNew' || tags['gc'] == 'G1 Young Generation') {tags.gc = 'young_gc_count'} })"); + assertNotNull(java); + + assertTrue(java.contains("if (\"PS Scavenge\".equals(tags.get(\"gc\"))"), + "Should translate first == with constant on left for null-safety"); + assertTrue(java.contains("|| \"Copy\".equals(tags.get(\"gc\"))"), + "Should chain || for second comparison"); + assertTrue(java.contains("|| \"ParNew\".equals(tags.get(\"gc\"))"), + "Should chain || for third comparison"); + assertTrue(java.contains("|| \"G1 Young Generation\".equals(tags.get(\"gc\"))"), + "Should chain || for fourth comparison"); + assertTrue(java.contains("tags.put(\"gc\", \"young_gc_count\")"), + "Should translate assignment in if body"); + } + + @Test + void ifElse() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['primary'] == 'true') {tags.primary = 'primary'} else {tags.primary = 'replica'} })"); + assertNotNull(java); + + assertTrue(java.contains("if (\"true\".equals(tags.get(\"primary\"))"), + "Should translate == comparison in condition"); + assertTrue(java.contains("tags.put(\"primary\", \"primary\")"), + "Should translate if-branch assignment"); + assertTrue(java.contains("} else {"), + "Should have else clause"); + assertTrue(java.contains("tags.put(\"primary\", \"replica\")"), + "Should translate else-branch assignment"); + } + + @Test + void ifOnlyNoElse() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['level'] == '1') {tags.level = 'L1 aggregation'} })"); + assertNotNull(java); + + assertTrue(java.contains("if (\"1\".equals(tags.get(\"level\"))"), + "Should translate condition"); + assertTrue(java.contains("tags.put(\"level\", \"L1 aggregation\")"), + "Should translate if-body"); + assertTrue(!java.contains("else"), + "Should NOT have else clause"); + } + + @Test + void notEqualComparison() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['status'] != 'ok') {tags.status = 'error'} })"); + assertNotNull(java); + + assertTrue(java.contains("!\"ok\".equals(tags.get(\"status\"))"), + "Should translate != with negated .equals()"); + } + + @Test + void logicalAndCondition() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['a'] == 'x' && tags['b'] == 'y') {tags.c = 'z'} })"); + assertNotNull(java); + + assertTrue(java.contains("\"x\".equals(tags.get(\"a\")) && \"y\".equals(tags.get(\"b\"))"), + "Should translate && with .equals() on both sides"); + } + + @Test + void chainedTagClosuresWithIf() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['level'] == '1') {tags.level = 'L1 aggregation'} })" + + ".tag({tags -> if (tags['level'] == '2') {tags.level = 'L2 aggregation'} })"); + assertNotNull(java); + + assertTrue(java.contains("\"1\".equals(tags.get(\"level\"))"), + "Should translate first tag closure condition"); + assertTrue(java.contains("\"2\".equals(tags.get(\"level\"))"), + "Should translate second tag closure condition"); + } + + @Test + void ifWithMethodChainAfter() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> if (tags['gc'] == 'Copy') {tags.gc = 'young'} }).sum(['host']).service(['svc'], Layer.GENERAL)"); + assertNotNull(java); + + assertTrue(java.contains(".tag((TagFunction)"), + "Should have TagFunction cast"); + assertTrue(java.contains("\"Copy\".equals(tags.get(\"gc\"))"), + "Should have if condition"); + assertTrue(java.contains(".sum(List.of(\"host\"))"), + "Should chain sum after tag"); + } + + // ---- Filter Closures ---- + + @Test + void simpleEqualityFilter() { + final String java = transpiler.transpileFilter("MalFilter_0", + "{ tags -> tags.job_name == 'vm-monitoring' }"); + assertNotNull(java); + + assertTrue(java.contains("public class MalFilter_0 implements MalFilter"), + "Should implement MalFilter"); + assertTrue(java.contains("public boolean test(Map tags)"), + "Should have test method"); + assertTrue(java.contains("\"vm-monitoring\".equals(tags.get(\"job_name\"))"), + "Should translate == with constant on left"); + } + + @Test + void filterPackageAndImports() { + final String java = transpiler.transpileFilter("MalFilter_0", + "{ tags -> tags.job_name == 'x' }"); + assertNotNull(java); + + assertTrue(java.contains("package " + MalToJavaTranspiler.GENERATED_PACKAGE), + "Should have correct package"); + assertTrue(java.contains("import java.util.*;"), + "Should import java.util"); + assertTrue(java.contains("import org.apache.skywalking.oap.meter.analyzer.dsl.*;"), + "Should import dsl package"); + } + + @Test + void orFilter() { + final String java = transpiler.transpileFilter("MalFilter_1", + "{ tags -> tags.job_name == 'flink-jobManager-monitoring' || tags.job_name == 'flink-taskManager-monitoring' }"); + assertNotNull(java); + + assertTrue(java.contains("\"flink-jobManager-monitoring\".equals(tags.get(\"job_name\"))"), + "Should translate first =="); + assertTrue(java.contains("|| \"flink-taskManager-monitoring\".equals(tags.get(\"job_name\"))"), + "Should translate || with second =="); + } + + @Test + void inListFilter() { + final String java = transpiler.transpileFilter("MalFilter_2", + "{ tags -> tags.job_name in ['kubernetes-cadvisor', 'kube-state-metrics'] }"); + assertNotNull(java); + + assertTrue(java.contains("List.of(\"kubernetes-cadvisor\", \"kube-state-metrics\").contains(tags.get(\"job_name\"))"), + "Should translate 'in' to List.of().contains()"); + } + + @Test + void compoundAndFilter() { + final String java = transpiler.transpileFilter("MalFilter_3", + "{ tags -> tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3' }"); + assertNotNull(java); + + assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), + "Should translate first =="); + assertTrue(java.contains("&& \"AWS/S3\".equals(tags.get(\"Namespace\"))"), + "Should translate && with second =="); + } + + @Test + void truthinessFilter() { + final String java = transpiler.transpileFilter("MalFilter_4", + "{ tags -> tags.cloud_provider == 'aws' && tags.Stage }"); + assertNotNull(java); + + assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), + "Should translate =="); + assertTrue(java.contains("(tags.get(\"Stage\") != null && !tags.get(\"Stage\").isEmpty())"), + "Should translate bare tags.Stage as truthiness check"); + } + + @Test + void negatedTruthinessFilter() { + final String java = transpiler.transpileFilter("MalFilter_5", + "{ tags -> tags.cloud_provider == 'aws' && !tags.Method }"); + assertNotNull(java); + + assertTrue(java.contains("(tags.get(\"Method\") == null || tags.get(\"Method\").isEmpty())"), + "Should translate !tags.Method as negated truthiness"); + } + + @Test + void compoundWithTruthinessAndNegation() { + final String java = transpiler.transpileFilter("MalFilter_6", + "{ tags -> tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/ApiGateway' && tags.Stage && !tags.Method }"); + assertNotNull(java); + + assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), + "Should translate first =="); + assertTrue(java.contains("\"AWS/ApiGateway\".equals(tags.get(\"Namespace\"))"), + "Should translate second =="); + assertTrue(java.contains("(tags.get(\"Stage\") != null && !tags.get(\"Stage\").isEmpty())"), + "Should translate truthiness"); + assertTrue(java.contains("(tags.get(\"Method\") == null || tags.get(\"Method\").isEmpty())"), + "Should translate negated truthiness"); + } + + @Test + void wrappedBlockFilter() { + final String java = transpiler.transpileFilter("MalFilter_7", + "{ tags -> {tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3'} }"); + assertNotNull(java); + + assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), + "Should unwrap inner block and translate =="); + assertTrue(java.contains("\"AWS/S3\".equals(tags.get(\"Namespace\"))"), + "Should translate second == after unwrapping"); + } + + @Test + void truthinessWithOrInParens() { + final String java = transpiler.transpileFilter("MalFilter_8", + "{ tags -> tags.cloud_provider == 'aws' && (tags.ApiId || tags.ApiName) }"); + assertNotNull(java); + + assertTrue(java.contains("(tags.get(\"ApiId\") != null && !tags.get(\"ApiId\").isEmpty())"), + "Should translate ApiId truthiness"); + assertTrue(java.contains("(tags.get(\"ApiName\") != null && !tags.get(\"ApiName\").isEmpty())"), + "Should translate ApiName truthiness"); + } + + // ---- forEach() Closure ---- + + @Test + void forEachBasicStructure() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['client', 'server'], { prefix, tags -> tags[prefix + '_id'] = 'test' })"); + assertNotNull(java); + + assertTrue(java.contains(".forEach(List.of(\"client\", \"server\"), (ForEachFunction)"), + "Should cast closure to ForEachFunction"); + assertTrue(java.contains("(prefix, tags) -> {"), + "Should have two-parameter lambda"); + assertTrue(java.contains("tags.put(prefix + \"_id\", \"test\")"), + "Should translate dynamic subscript write"); + } + + @Test + void forEachNullCheckWithEarlyReturn() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['client'], { prefix, tags -> if (tags[prefix + '_process_id'] != null) { return } })"); + assertNotNull(java); + + assertTrue(java.contains("tags.get(prefix + \"_process_id\") != null"), + "Should translate null check"); + assertTrue(java.contains("return;"), + "Should have void return for early exit"); + } + + @Test + void forEachWithProcessRegistry() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['client'], { prefix, tags -> " + + "tags[prefix + '_process_id'] = ProcessRegistry.generateVirtualLocalProcess(tags.service, tags.instance) })"); + assertNotNull(java); + + assertTrue(java.contains("ProcessRegistry.generateVirtualLocalProcess(tags.get(\"service\"), tags.get(\"instance\"))"), + "Should translate static method call with tag reads as args"); + assertTrue(java.contains("tags.put(prefix + \"_process_id\","), + "Should translate dynamic subscript write"); + } + + @Test + void forEachVarDeclaration() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['component'], { key, tags -> String result = '' })"); + assertNotNull(java); + + assertTrue(java.contains("String result = \"\""), + "Should translate variable declaration with empty string"); + } + + @Test + void forEachVarDeclWithTagRead() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['component'], { key, tags -> String protocol = tags['protocol'] })"); + assertNotNull(java); + + assertTrue(java.contains("String protocol = tags.get(\"protocol\")"), + "Should translate var decl with tag read"); + } + + @Test + void forEachIfElseIfChain() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['component'], { key, tags -> " + + "String protocol = tags['protocol']\n" + + "String ssl = tags['is_ssl']\n" + + "String result = ''\n" + + "if (protocol == 'http' && ssl == 'true') { result = '129' } " + + "else if (protocol == 'http') { result = '49' } " + + "else if (ssl == 'true') { result = '130' } " + + "else { result = '110' }\n" + + "tags[key] = result })"); + assertNotNull(java); + + assertTrue(java.contains("String protocol = tags.get(\"protocol\")"), + "Should declare protocol"); + assertTrue(java.contains("String ssl = tags.get(\"is_ssl\")"), + "Should declare ssl"); + + assertTrue(java.contains("\"http\".equals(protocol)"), + "Should compare local var with .equals()"); + assertTrue(java.contains("\"true\".equals(ssl)"), + "Should compare ssl with .equals()"); + + assertTrue(java.contains("} else if ("), + "Should produce else-if, not nested else { if }"); + + assertTrue(java.contains("} else {"), + "Should have final else"); + assertTrue(java.contains("result = \"110\""), + "Should assign default value in else"); + + assertTrue(java.contains("tags.put(key, result)"), + "Should write result to tags[key]"); + } + + @Test + void forEachLocalVarAssignment() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['x'], { key, tags -> " + + "String r = ''\n" + + "r = 'abc'\n" + + "tags[key] = r })"); + assertNotNull(java); + + assertTrue(java.contains("r = \"abc\""), + "Should translate local var reassignment"); + } + + @Test + void forEachEqualsOnStringComparison() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['client'], { prefix, tags -> " + + "if (tags[prefix + '_local'] == 'true') { tags[prefix + '_id'] = 'local' } })"); + assertNotNull(java); + + assertTrue(java.contains("\"true\".equals(tags.get(prefix + \"_local\"))"), + "Should translate dynamic subscript comparison with .equals()"); + assertTrue(java.contains("tags.put(prefix + \"_id\", \"local\")"), + "Should translate dynamic subscript assignment"); + } + + @Test + void chainedForEach() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.forEach(['a'], { k1, tags -> tags[k1] = 'x' })" + + ".forEach(['b'], { k2, tags -> tags[k2] = 'y' })"); + assertNotNull(java); + + assertTrue(java.contains("(ForEachFunction) (k1, tags)"), + "Should have first forEach with k1"); + assertTrue(java.contains("(ForEachFunction) (k2, tags)"), + "Should have second forEach with k2"); + } + + // ---- Elvis (?:), Safe Navigation (?.), Ternary (? :) ---- + + @Test + void safeNavigation() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.svc = tags['skywalking_service']?.trim() })"); + assertNotNull(java); + + assertTrue(java.contains("(tags.get(\"skywalking_service\") != null ? tags.get(\"skywalking_service\").trim() : null)"), + "Should translate ?.trim() to null-checked call"); + } + + @Test + void elvisOperator() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.svc = tags['name'] ?: 'unknown' })"); + assertNotNull(java); + + assertTrue(java.contains("(tags.get(\"name\") != null ? tags.get(\"name\") : \"unknown\")"), + "Should translate ?: to null-check with default"); + } + + @Test + void safeNavPlusElvis() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.service_name = 'APISIX::'+(tags['skywalking_service']?.trim()?:'APISIX') })"); + assertNotNull(java); + + assertTrue(java.contains("tags.get(\"skywalking_service\") != null ? tags.get(\"skywalking_service\").trim() : null"), + "Should have safe nav for trim"); + assertTrue(java.contains("!= null ?") && java.contains(": \"APISIX\""), + "Should have elvis default to APISIX"); + assertTrue(java.contains("\"APISIX::\" + "), + "Should have string prefix concatenation"); + } + + @Test + void ternaryOperator() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.tag({tags -> tags.service_name = tags.ApiId ? 'gw::'+tags.ApiId : 'gw::'+tags.ApiName })"); + assertNotNull(java); + + assertTrue(java.contains("tags.get(\"ApiId\") != null && !tags.get(\"ApiId\").isEmpty()"), + "Should translate ternary condition as truthiness check"); + assertTrue(java.contains("\"gw::\" + tags.get(\"ApiId\")"), + "Should have true branch expression"); + assertTrue(java.contains("\"gw::\" + tags.get(\"ApiName\")"), + "Should have false branch expression"); + } + + @Test + void safeNavInFilterCondition() { + final String java = transpiler.transpileFilter("MalFilter_test", + "{ tags -> tags.job_name == 'eks-monitoring' && tags.Service?.trim() }"); + assertNotNull(java); + + assertTrue(java.contains("\"eks-monitoring\".equals(tags.get(\"job_name\"))"), + "Should translate == comparison"); + assertTrue(java.contains("tags.get(\"Service\") != null ? tags.get(\"Service\").trim() : null"), + "Should have safe nav for Service?.trim()"); + } + + // ---- instance() with PropertiesExtractor, MapExpression ---- + + @Test + void instanceWithPropertiesExtractor() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, " + + "{tags -> ['pod': tags.pod, 'namespace': tags.namespace]})"); + assertNotNull(java); + + assertTrue(java.contains(".instance("), + "Should have instance call"); + assertTrue(java.contains("(PropertiesExtractor)"), + "Should cast closure to PropertiesExtractor"); + assertTrue(java.contains("Map.of(\"pod\", tags.get(\"pod\"), \"namespace\", tags.get(\"namespace\"))"), + "Should translate map literal to Map.of()"); + } + + @Test + void mapExpressionInTagValue() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.instance(['svc'], ['inst'], Layer.GENERAL, {tags -> ['key': tags.val]})"); + assertNotNull(java); + + assertTrue(java.contains("Map.of(\"key\", tags.get(\"val\"))"), + "Should translate single-entry map"); + assertTrue(java.contains("(PropertiesExtractor)"), + "Should have PropertiesExtractor cast"); + } + + @Test + void processRelationNoClosures() { + final String java = transpiler.transpileExpression("MalExpr_test", + "metric.processRelation('side', ['service'], ['instance'], " + + "'client_process_id', 'server_process_id', 'component')"); + assertNotNull(java); + + assertTrue(java.contains(".processRelation(\"side\", List.of(\"service\"), List.of(\"instance\"), " + + "\"client_process_id\", \"server_process_id\", \"component\")"), + "Should translate processRelation as regular method call"); + } + + // ---- Compilation + Manifests ---- + + @Test + void sourceWrittenForCompilation(@TempDir Path tempDir) throws Exception { + final String source = transpiler.transpileExpression("MalExpr_compile_test", + "metric.sum(['host']).service(['svc'], Layer.GENERAL)"); + transpiler.registerExpression("MalExpr_compile_test", source); + + final File sourceDir = tempDir.resolve("src").toFile(); + final File outputDir = tempDir.resolve("classes").toFile(); + + try { + transpiler.compileAll(sourceDir, outputDir, System.getProperty("java.class.path")); + + final String classPath = MalToJavaTranspiler.GENERATED_PACKAGE.replace('.', File.separatorChar) + + File.separator + "MalExpr_compile_test.class"; + assertTrue(new File(outputDir, classPath).exists(), + "Compiled .class file should exist"); + } catch (RuntimeException e) { + if (e.getMessage().contains("compilation failed")) { + final String pkgPath = MalToJavaTranspiler.GENERATED_PACKAGE.replace('.', File.separatorChar); + final File javaFile = new File(sourceDir, pkgPath + "/MalExpr_compile_test.java"); + assertTrue(javaFile.exists(), "Source .java file should be written"); + final String written = Files.readString(javaFile.toPath()); + assertTrue(written.contains("implements MalExpression"), + "Written source should implement MalExpression"); + } else { + throw e; + } + } + } + + @Test + void expressionManifest(@TempDir Path tempDir) throws Exception { + transpiler.registerExpression("MalExpr_a", + transpiler.transpileExpression("MalExpr_a", "metric_a")); + transpiler.registerExpression("MalExpr_b", + transpiler.transpileExpression("MalExpr_b", "metric_b")); + + final File outputDir = tempDir.toFile(); + transpiler.writeExpressionManifest(outputDir); + + final File manifest = new File(outputDir, "META-INF/mal-expressions.txt"); + assertTrue(manifest.exists(), "Manifest file should exist"); + + final List lines = Files.readAllLines(manifest.toPath()); + assertTrue(lines.contains(MalToJavaTranspiler.GENERATED_PACKAGE + ".MalExpr_a"), + "Should contain MalExpr_a FQCN"); + assertTrue(lines.contains(MalToJavaTranspiler.GENERATED_PACKAGE + ".MalExpr_b"), + "Should contain MalExpr_b FQCN"); + } + + @Test + void filterManifest(@TempDir Path tempDir) throws Exception { + final String literal = "{ tags -> tags.job == 'x' }"; + transpiler.registerFilter("MalFilter_0", literal, + transpiler.transpileFilter("MalFilter_0", literal)); + + final File outputDir = tempDir.toFile(); + transpiler.writeFilterManifest(outputDir); + + final File manifest = new File(outputDir, "META-INF/mal-filter-expressions.properties"); + assertTrue(manifest.exists(), "Filter manifest should exist"); + + final String content = Files.readString(manifest.toPath()); + assertTrue(content.contains(MalToJavaTranspiler.GENERATED_PACKAGE + ".MalFilter_0"), + "Should contain MalFilter_0 FQCN"); + } +} diff --git a/oap-server/analyzer/pom.xml b/oap-server/analyzer/pom.xml index 9dca94257fea..4039928a5023 100644 --- a/oap-server/analyzer/pom.xml +++ b/oap-server/analyzer/pom.xml @@ -33,6 +33,7 @@ log-analyzer meter-analyzer event-analyzer + mal-transpiler From a22f67a7918a711b0bf38a66775bdb9689168568 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 15:42:37 +0800 Subject: [PATCH 04/64] Add LAL transpiler module for build-time Groovy-to-Java conversion (Phase 3) Introduces lal-transpiler module that parses LAL Groovy DSL scripts into AST at Phases.CONVERSION and emits pure Java classes implementing LalExpression. Handles filter/text/json/yaml/extractor/sink/abort blocks, parsed property access, safe navigation, cast expressions, GString interpolation, and SHA-256 deduplication. Makes MalToJavaTranspiler.escapeJava() public for cross-module reuse. Includes 37 comprehensive tests. Co-Authored-By: Claude Opus 4.6 --- oap-server/analyzer/lal-transpiler/pom.xml | 46 + .../transpiler/lal/LalToJavaTranspiler.java | 923 ++++++++++++++++++ .../lal/LalToJavaTranspilerTest.java | 678 +++++++++++++ .../transpiler/mal/MalToJavaTranspiler.java | 2 +- oap-server/analyzer/pom.xml | 1 + 5 files changed, 1649 insertions(+), 1 deletion(-) create mode 100644 oap-server/analyzer/lal-transpiler/pom.xml create mode 100644 oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java create mode 100644 oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java diff --git a/oap-server/analyzer/lal-transpiler/pom.xml b/oap-server/analyzer/lal-transpiler/pom.xml new file mode 100644 index 000000000000..b722b255fcfc --- /dev/null +++ b/oap-server/analyzer/lal-transpiler/pom.xml @@ -0,0 +1,46 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + lal-transpiler + + + + org.apache.skywalking + log-analyzer + ${project.version} + + + org.apache.skywalking + mal-transpiler + ${project.version} + + + org.apache.groovy + groovy + + + diff --git a/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java b/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java new file mode 100644 index 000000000000..ae7ad63f088e --- /dev/null +++ b/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java @@ -0,0 +1,923 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.transpiler.lal; + +import java.io.File; +import java.io.IOException; +import java.io.StringWriter; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; +import javax.tools.JavaCompiler; +import javax.tools.JavaFileObject; +import javax.tools.StandardJavaFileManager; +import javax.tools.ToolProvider; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.server.transpiler.mal.MalToJavaTranspiler; +import org.codehaus.groovy.ast.ModuleNode; +import org.codehaus.groovy.ast.expr.ArgumentListExpression; +import org.codehaus.groovy.ast.expr.BinaryExpression; +import org.codehaus.groovy.ast.expr.BooleanExpression; +import org.codehaus.groovy.ast.expr.CastExpression; +import org.codehaus.groovy.ast.expr.ClosureExpression; +import org.codehaus.groovy.ast.expr.ConstantExpression; +import org.codehaus.groovy.ast.expr.Expression; +import org.codehaus.groovy.ast.expr.GStringExpression; +import org.codehaus.groovy.ast.expr.MapEntryExpression; +import org.codehaus.groovy.ast.expr.MapExpression; +import org.codehaus.groovy.ast.expr.MethodCallExpression; +import org.codehaus.groovy.ast.expr.NotExpression; +import org.codehaus.groovy.ast.expr.PropertyExpression; +import org.codehaus.groovy.ast.expr.TupleExpression; +import org.codehaus.groovy.ast.expr.VariableExpression; +import org.codehaus.groovy.ast.stmt.BlockStatement; +import org.codehaus.groovy.ast.stmt.EmptyStatement; +import org.codehaus.groovy.ast.stmt.ExpressionStatement; +import org.codehaus.groovy.ast.stmt.IfStatement; +import org.codehaus.groovy.ast.stmt.Statement; +import org.codehaus.groovy.control.CompilationUnit; +import org.codehaus.groovy.control.CompilerConfiguration; +import org.codehaus.groovy.control.Phases; +import org.codehaus.groovy.syntax.Types; + +/** + * Transpiles LAL (Log Analysis Language) Groovy scripts to Java source code at build time. + * Parses DSL strings into Groovy AST (via CompilationUnit at CONVERSION phase), + * walks AST nodes, and produces Java classes implementing LalExpression. + */ +@Slf4j +public class LalToJavaTranspiler { + + static final String GENERATED_PACKAGE = + "org.apache.skywalking.oap.server.core.source.oal.rt.lal"; + + private static final Set CONSUMER_METHODS = Set.of( + "text", "extractor", "sink", "slowSql", "sampledTrace", "metrics", "sampler" + ); + + private static final Set STATIC_TYPES = Set.of("ProcessRegistry"); + + // ---- Batch state ---- + + private final Map lalSources = new LinkedHashMap<>(); + private final Map hashToClass = new LinkedHashMap<>(); + + private int tempVarCounter; + + /** + * Transpile a LAL DSL script to a Java class implementing LalExpression. + * + * @param className simple class name (e.g. "LalExpr_3") + * @param dslText the Groovy DSL string (filter { ... }) + * @return Java source code + */ + public String transpile(final String className, final String dslText) { + tempVarCounter = 0; + final ModuleNode module = parseToAST(dslText); + final Statement body = extractBody(module); + + final StringBuilder sb = new StringBuilder(); + emitHeader(sb, className); + + final List stmts = flattenStatements(body); + for (Statement stmt : stmts) { + emitStatement(sb, stmt, "filterSpec", "binding", 2); + } + + emitFooter(sb); + return sb.toString(); + } + + public void register(final String className, final String hash, final String source) { + lalSources.put(className, source); + hashToClass.put(hash, GENERATED_PACKAGE + "." + className); + } + + // ---- AST Parsing ---- + + ModuleNode parseToAST(final String expression) { + final CompilerConfiguration cc = new CompilerConfiguration(); + final CompilationUnit cu = new CompilationUnit(cc); + cu.addSource("Script", expression); + cu.compile(Phases.CONVERSION); + final List modules = cu.getAST().getModules(); + if (modules.isEmpty()) { + throw new IllegalStateException("No AST modules produced"); + } + return modules.get(0); + } + + Statement extractBody(final ModuleNode module) { + final BlockStatement block = module.getStatementBlock(); + if (block != null && !block.getStatements().isEmpty()) { + return block; + } + throw new IllegalStateException("Empty AST body"); + } + + // ---- Code Generation ---- + + private void emitHeader(final StringBuilder sb, final String className) { + sb.append("package ").append(GENERATED_PACKAGE).append(";\n\n"); + sb.append("import java.util.Map;\n"); + sb.append("import org.apache.skywalking.oap.log.analyzer.dsl.Binding;\n"); + sb.append("import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression;\n"); + sb.append("import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec;\n\n"); + sb.append("@SuppressWarnings(\"unchecked\")\n"); + sb.append("public class ").append(className).append(" implements LalExpression {\n\n"); + emitHelpers(sb); + sb.append(" @Override\n"); + sb.append(" public void execute(FilterSpec filterSpec, Binding binding) {\n"); + } + + private void emitHelpers(final StringBuilder sb) { + sb.append(" private static Object getAt(Object obj, String key) {\n"); + sb.append(" if (obj == null) return null;\n"); + sb.append(" if (obj instanceof Binding.Parsed) return ((Binding.Parsed) obj).getAt(key);\n"); + sb.append(" if (obj instanceof Map) return ((Map) obj).get(key);\n"); + sb.append(" return null;\n"); + sb.append(" }\n\n"); + + sb.append(" private static long toLong(Object obj) {\n"); + sb.append(" if (obj instanceof Number) return ((Number) obj).longValue();\n"); + sb.append(" if (obj instanceof String) return Long.parseLong((String) obj);\n"); + sb.append(" return 0L;\n"); + sb.append(" }\n\n"); + + sb.append(" private static int toInt(Object obj) {\n"); + sb.append(" if (obj instanceof Number) return ((Number) obj).intValue();\n"); + sb.append(" if (obj instanceof String) return Integer.parseInt((String) obj);\n"); + sb.append(" return 0;\n"); + sb.append(" }\n\n"); + + sb.append(" private static boolean toBoolean(Object obj) {\n"); + sb.append(" if (obj instanceof Boolean) return (Boolean) obj;\n"); + sb.append(" if (obj instanceof String) return Boolean.parseBoolean((String) obj);\n"); + sb.append(" return obj != null;\n"); + sb.append(" }\n\n"); + + sb.append(" private static boolean isTruthy(Object obj) {\n"); + sb.append(" if (obj == null) return false;\n"); + sb.append(" if (obj instanceof Boolean) return (Boolean) obj;\n"); + sb.append(" if (obj instanceof String) return !((String) obj).isEmpty();\n"); + sb.append(" if (obj instanceof Number) return ((Number) obj).doubleValue() != 0;\n"); + sb.append(" return true;\n"); + sb.append(" }\n\n"); + + sb.append(" private static boolean isNonEmptyString(Object obj) {\n"); + sb.append(" if (obj == null) return false;\n"); + sb.append(" String s = obj.toString();\n"); + sb.append(" return s != null && !s.trim().isEmpty();\n"); + sb.append(" }\n\n"); + } + + private void emitFooter(final StringBuilder sb) { + sb.append(" }\n"); + sb.append("}\n"); + } + + private void emitStatement(final StringBuilder sb, final Statement stmt, + final String receiver, final String bindingVar, final int indent) { + if (stmt instanceof BlockStatement) { + for (Statement s : ((BlockStatement) stmt).getStatements()) { + emitStatement(sb, s, receiver, bindingVar, indent); + } + } else if (stmt instanceof ExpressionStatement) { + emitExpressionStatement(sb, ((ExpressionStatement) stmt).getExpression(), + receiver, bindingVar, indent); + } else if (stmt instanceof IfStatement) { + emitIfStatement(sb, (IfStatement) stmt, receiver, bindingVar, indent); + } else if (stmt instanceof EmptyStatement) { + // skip + } else { + throw new UnsupportedOperationException( + "Unsupported statement: " + stmt.getClass().getSimpleName()); + } + } + + private void emitExpressionStatement(final StringBuilder sb, final Expression expr, + final String receiver, final String bindingVar, + final int indent) { + if (expr instanceof MethodCallExpression) { + emitMethodCall(sb, (MethodCallExpression) expr, receiver, bindingVar, indent); + } else if (expr instanceof BinaryExpression) { + emitBinaryAsNamedArg(sb, (BinaryExpression) expr, receiver, bindingVar, indent); + } else { + throw new UnsupportedOperationException( + "Unsupported expression statement: " + expr.getClass().getSimpleName() + + " (" + expr.getText() + ")"); + } + } + + private void emitMethodCall(final StringBuilder sb, final MethodCallExpression mce, + final String receiver, final String bindingVar, final int indent) { + final String methodName = mce.getMethodAsString(); + final Expression objExpr = mce.getObjectExpression(); + final ArgumentListExpression args = toArgList(mce.getArguments()); + final List argExprs = args.getExpressions(); + + // Top-level filter { ... } -> unwrap and emit body + if ("filter".equals(methodName) && isThisOrImplicit(objExpr)) { + if (!argExprs.isEmpty() && argExprs.get(0) instanceof ClosureExpression) { + final ClosureExpression closure = (ClosureExpression) argExprs.get(0); + emitStatement(sb, closure.getCode(), receiver, bindingVar, indent); + return; + } + } + + // json {} -> filterSpec.json() + if ("json".equals(methodName) && isThisOrImplicit(objExpr)) { + indent(sb, indent); + sb.append(receiver).append(".json();\n"); + return; + } + + // text { regexp /pattern/ } -> filterSpec.text(tp -> tp.regexp("pattern")) + if ("text".equals(methodName) && isThisOrImplicit(objExpr) + && !argExprs.isEmpty() && argExprs.get(0) instanceof ClosureExpression) { + final ClosureExpression closure = (ClosureExpression) argExprs.get(0); + final String tpVar = "tp"; + indent(sb, indent); + sb.append(receiver).append(".text(").append(tpVar).append(" -> {\n"); + emitStatement(sb, closure.getCode(), tpVar, bindingVar, indent + 1); + indent(sb, indent); + sb.append("});\n"); + return; + } + + // regexp /pattern/ -> tp.regexp("pattern") + if ("regexp".equals(methodName) && isThisOrImplicit(objExpr)) { + indent(sb, indent); + sb.append(receiver).append(".regexp("); + if (!argExprs.isEmpty()) { + sb.append(visitValueExpression(argExprs.get(0), bindingVar)); + } + sb.append(");\n"); + return; + } + + // Consumer overload methods: extractor, sink, slowSql, sampledTrace, metrics, sampler + if (CONSUMER_METHODS.contains(methodName) && isThisOrImplicit(objExpr) + && !argExprs.isEmpty() && argExprs.get(0) instanceof ClosureExpression) { + final ClosureExpression closure = (ClosureExpression) argExprs.get(0); + final String lambdaVar = lambdaVarFor(methodName); + final String childReceiver = childReceiverFor(lambdaVar); + indent(sb, indent); + sb.append(receiver).append(".").append(methodName).append("(") + .append(lambdaVar).append(" -> {\n"); + emitStatement(sb, closure.getCode(), childReceiver, bindingVar, indent + 1); + indent(sb, indent); + sb.append("});\n"); + return; + } + + // rateLimit("${expr}") { rpm N } -> sp.rateLimit(idExpr, rls -> rls.rpm(N)) + if ("rateLimit".equals(methodName) && isThisOrImplicit(objExpr)) { + final Expression idArg = argExprs.get(0); + final ClosureExpression closure = argExprs.size() > 1 + && argExprs.get(1) instanceof ClosureExpression + ? (ClosureExpression) argExprs.get(1) : null; + + indent(sb, indent); + sb.append(receiver).append(".rateLimit(") + .append(visitValueExpression(idArg, bindingVar)); + if (closure != null) { + final String rlsVar = "rls"; + sb.append(", ").append(rlsVar).append(" -> {\n"); + emitStatement(sb, closure.getCode(), rlsVar, bindingVar, indent + 1); + indent(sb, indent); + sb.append("}"); + } + sb.append(");\n"); + return; + } + + // abort {} -> filterSpec.abort() + if ("abort".equals(methodName) && isThisOrImplicit(objExpr)) { + indent(sb, indent); + sb.append(receiver).append(".abort();\n"); + return; + } + + // enforcer {} or dropper {} -> sink.enforcer() / sink.dropper() + if (("enforcer".equals(methodName) || "dropper".equals(methodName)) + && isThisOrImplicit(objExpr)) { + indent(sb, indent); + sb.append(receiver).append(".").append(methodName).append("();\n"); + return; + } + + // tag("KEY") as standalone statement + if ("tag".equals(methodName) && isThisOrImplicit(objExpr) + && !argExprs.isEmpty() && argExprs.get(0) instanceof ConstantExpression) { + indent(sb, indent); + sb.append(receiver).append(".tag(\"") + .append(MalToJavaTranspiler.escapeJava(argExprs.get(0).getText())) + .append("\");\n"); + return; + } + + // Simple value-setting methods: service(val), layer(val), timestamp(val), etc. + if (isThisOrImplicit(objExpr)) { + indent(sb, indent); + sb.append(receiver).append(".").append(methodName).append("("); + for (int i = 0; i < argExprs.size(); i++) { + if (i > 0) { + sb.append(", "); + } + sb.append(visitValueExpression(argExprs.get(i), bindingVar)); + } + sb.append(");\n"); + return; + } + + // Static method: ProcessRegistry.generateVirtualLocalProcess(...) + if (objExpr instanceof VariableExpression + && STATIC_TYPES.contains(((VariableExpression) objExpr).getName())) { + indent(sb, indent); + sb.append("org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry.") + .append(methodName).append("("); + for (int i = 0; i < argExprs.size(); i++) { + if (i > 0) { + sb.append(", "); + } + sb.append(visitValueExpression(argExprs.get(i), bindingVar)); + } + sb.append(");\n"); + return; + } + + throw new UnsupportedOperationException( + "Unsupported method call: " + methodName + " on " + objExpr.getClass().getSimpleName() + + " (" + mce.getText() + ")"); + } + + private void emitBinaryAsNamedArg(final StringBuilder sb, final BinaryExpression expr, + final String receiver, final String bindingVar, + final int indent) { + throw new UnsupportedOperationException( + "Unsupported binary expression as statement: " + expr.getText()); + } + + // ---- If/Else ---- + + private void emitIfStatement(final StringBuilder sb, final IfStatement ifStmt, + final String receiver, final String bindingVar, final int indent) { + indent(sb, indent); + sb.append("if ("); + sb.append(visitCondition(ifStmt.getBooleanExpression(), bindingVar)); + sb.append(") {\n"); + + emitStatement(sb, ifStmt.getIfBlock(), receiver, bindingVar, indent + 1); + + final Statement elseBlock = ifStmt.getElseBlock(); + if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { + indent(sb, indent); + if (elseBlock instanceof IfStatement) { + sb.append("} else "); + emitIfStatementInline(sb, (IfStatement) elseBlock, receiver, bindingVar, indent); + return; + } else { + sb.append("} else {\n"); + emitStatement(sb, elseBlock, receiver, bindingVar, indent + 1); + indent(sb, indent); + sb.append("}\n"); + } + } else { + indent(sb, indent); + sb.append("}\n"); + } + } + + private void emitIfStatementInline(final StringBuilder sb, final IfStatement ifStmt, + final String receiver, final String bindingVar, + final int indent) { + sb.append("if ("); + sb.append(visitCondition(ifStmt.getBooleanExpression(), bindingVar)); + sb.append(") {\n"); + + emitStatement(sb, ifStmt.getIfBlock(), receiver, bindingVar, indent + 1); + + final Statement elseBlock = ifStmt.getElseBlock(); + if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { + indent(sb, indent); + if (elseBlock instanceof IfStatement) { + sb.append("} else "); + emitIfStatementInline(sb, (IfStatement) elseBlock, receiver, bindingVar, indent); + } else { + sb.append("} else {\n"); + emitStatement(sb, elseBlock, receiver, bindingVar, indent + 1); + indent(sb, indent); + sb.append("}\n"); + } + } else { + indent(sb, indent); + sb.append("}\n"); + } + } + + // ---- Condition Visiting ---- + + private String visitCondition(final BooleanExpression boolExpr, final String bindingVar) { + return visitConditionExpr(boolExpr.getExpression(), bindingVar); + } + + String visitConditionExpr(final Expression expr, final String bindingVar) { + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + final int op = bin.getOperation().getType(); + + if (op == Types.COMPARE_EQUAL) { + final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); + final String right = visitValueExpression(bin.getRightExpression(), bindingVar); + if (bin.getRightExpression() instanceof ConstantExpression + && ((ConstantExpression) bin.getRightExpression()).getValue() instanceof String) { + return "\"" + MalToJavaTranspiler.escapeJava( + (String) ((ConstantExpression) bin.getRightExpression()).getValue()) + + "\".equals(" + left + ")"; + } + return "java.util.Objects.equals(" + left + ", " + right + ")"; + } + + if (op == Types.COMPARE_NOT_EQUAL) { + final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); + final String right = visitValueExpression(bin.getRightExpression(), bindingVar); + if (bin.getRightExpression() instanceof ConstantExpression + && ((ConstantExpression) bin.getRightExpression()).getValue() instanceof String) { + return "!\"" + MalToJavaTranspiler.escapeJava( + (String) ((ConstantExpression) bin.getRightExpression()).getValue()) + + "\".equals(" + left + ")"; + } + return "!java.util.Objects.equals(" + left + ", " + right + ")"; + } + + if (op == Types.COMPARE_LESS_THAN) { + final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); + final String right = visitValueExpression(bin.getRightExpression(), bindingVar); + return "toInt(" + left + ") < " + right; + } + + if (op == Types.COMPARE_GREATER_THAN_EQUAL) { + final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); + final String right = visitValueExpression(bin.getRightExpression(), bindingVar); + return "toInt(" + left + ") >= " + right; + } + + if (op == Types.LOGICAL_AND) { + return visitConditionExpr(bin.getLeftExpression(), bindingVar) + + " && " + visitConditionExpr(bin.getRightExpression(), bindingVar); + } + + if (op == Types.LOGICAL_OR) { + return visitConditionExpr(bin.getLeftExpression(), bindingVar) + + " || " + visitConditionExpr(bin.getRightExpression(), bindingVar); + } + + throw new UnsupportedOperationException( + "Unsupported condition operator: " + bin.getOperation().getText()); + } + + if (expr instanceof NotExpression) { + final String inner = visitConditionExpr(((NotExpression) expr).getExpression(), bindingVar); + return "!" + inner; + } + + if (expr instanceof BooleanExpression) { + return visitConditionExpr(((BooleanExpression) expr).getExpression(), bindingVar); + } + + if (expr instanceof MethodCallExpression) { + final MethodCallExpression mce = (MethodCallExpression) expr; + final String methodName = mce.getMethodAsString(); + if ("toString".equals(methodName) || "trim".equals(methodName)) { + final String obj = visitValueExpression(mce.getObjectExpression(), bindingVar); + return "isNonEmptyString(" + obj + ")"; + } + return visitValueExpression(expr, bindingVar); + } + + return "isTruthy(" + visitValueExpression(expr, bindingVar) + ")"; + } + + // ---- Value Expression Visiting ---- + + String visitValueExpression(final Expression expr, final String bindingVar) { + if (expr instanceof ConstantExpression) { + return visitConstant((ConstantExpression) expr); + } + + if (expr instanceof VariableExpression) { + return visitVariable((VariableExpression) expr, bindingVar); + } + + if (expr instanceof PropertyExpression) { + return visitProperty((PropertyExpression) expr, bindingVar); + } + + if (expr instanceof CastExpression) { + return visitCast((CastExpression) expr, bindingVar); + } + + if (expr instanceof MethodCallExpression) { + return visitMethodCallValue((MethodCallExpression) expr, bindingVar); + } + + if (expr instanceof GStringExpression) { + return visitGString((GStringExpression) expr, bindingVar); + } + + if (expr instanceof MapExpression) { + return visitMapExpression((MapExpression) expr, bindingVar); + } + + if (expr instanceof BinaryExpression) { + final BinaryExpression bin = (BinaryExpression) expr; + final int op = bin.getOperation().getType(); + if (op == Types.COMPARE_EQUAL || op == Types.COMPARE_NOT_EQUAL + || op == Types.LOGICAL_AND || op == Types.LOGICAL_OR + || op == Types.COMPARE_LESS_THAN || op == Types.COMPARE_GREATER_THAN_EQUAL) { + return visitConditionExpr(expr, bindingVar); + } + } + + if (expr instanceof NotExpression) { + return "!" + visitValueExpression(((NotExpression) expr).getExpression(), bindingVar); + } + + throw new UnsupportedOperationException( + "Unsupported value expression: " + expr.getClass().getSimpleName() + + " (" + expr.getText() + ")"); + } + + private String visitConstant(final ConstantExpression expr) { + final Object value = expr.getValue(); + if (value instanceof String) { + return "\"" + MalToJavaTranspiler.escapeJava((String) value) + "\""; + } + if (value instanceof Integer) { + return value.toString(); + } + if (value instanceof Long) { + return value + "L"; + } + if (value instanceof Boolean) { + return value.toString(); + } + if (value instanceof Double) { + return value + "d"; + } + if (value == null) { + return "null"; + } + return value.toString(); + } + + private String visitVariable(final VariableExpression expr, final String bindingVar) { + final String name = expr.getName(); + if ("parsed".equals(name)) { + return bindingVar + ".parsed()"; + } + if ("log".equals(name)) { + return bindingVar + ".log()"; + } + if ("this".equals(name)) { + return "filterSpec"; + } + if (STATIC_TYPES.contains(name)) { + return "org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry"; + } + return name; + } + + private String visitProperty(final PropertyExpression expr, final String bindingVar) { + final Expression objExpr = expr.getObjectExpression(); + final String propName = expr.getPropertyAsString(); + final boolean isSafe = expr.isSafe(); + + final String obj = visitValueExpression(objExpr, bindingVar); + + // log.service -> binding.log().getService() + if (objExpr instanceof VariableExpression + && "log".equals(((VariableExpression) objExpr).getName())) { + return visitLogProperty(propName, bindingVar); + } + + // For parsed access and nested map access, use getAt() + if (isSafe) { + return "(" + obj + " == null ? null : getAt(" + obj + ", \"" + propName + "\"))"; + } + + return "getAt(" + obj + ", \"" + propName + "\")"; + } + + private String visitLogProperty(final String propName, final String bindingVar) { + switch (propName) { + case "service": + return bindingVar + ".log().getService()"; + case "serviceInstance": + return bindingVar + ".log().getServiceInstance()"; + case "endpoint": + return bindingVar + ".log().getEndpoint()"; + case "timestamp": + return bindingVar + ".log().getTimestamp()"; + default: + return bindingVar + ".log().get" + capitalize(propName) + "()"; + } + } + + private String visitCast(final CastExpression expr, final String bindingVar) { + final String inner = visitValueExpression(expr.getExpression(), bindingVar); + final String typeName = expr.getType().getName(); + + switch (typeName) { + case "java.lang.String": + case "String": + return "String.valueOf(" + inner + ")"; + case "java.lang.Long": + case "Long": + case "long": + return "toLong(" + inner + ")"; + case "java.lang.Integer": + case "Integer": + case "int": + return "toInt(" + inner + ")"; + case "java.lang.Boolean": + case "Boolean": + case "boolean": + return "toBoolean(" + inner + ")"; + default: + return "((" + typeName + ") " + inner + ")"; + } + } + + private String visitMethodCallValue(final MethodCallExpression mce, final String bindingVar) { + final String methodName = mce.getMethodAsString(); + final Expression objExpr = mce.getObjectExpression(); + final ArgumentListExpression args = toArgList(mce.getArguments()); + final boolean isSafe = mce.isSafe(); + + // tag("KEY") on filterSpec -> filterSpec.tag("KEY") + if ("tag".equals(methodName) && isThisOrImplicit(objExpr)) { + final List argExprs = args.getExpressions(); + if (!argExprs.isEmpty() && argExprs.get(0) instanceof ConstantExpression) { + return "filterSpec.tag(\"" + + MalToJavaTranspiler.escapeJava(argExprs.get(0).getText()) + "\")"; + } + } + + // ProcessRegistry.generateVirtualLocalProcess(...) + if (objExpr instanceof VariableExpression + && STATIC_TYPES.contains(((VariableExpression) objExpr).getName())) { + final StringBuilder sb = new StringBuilder(); + sb.append("org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry."); + sb.append(methodName).append("("); + final List argExprs = args.getExpressions(); + for (int i = 0; i < argExprs.size(); i++) { + if (i > 0) { + sb.append(", "); + } + sb.append(visitValueExpression(argExprs.get(i), bindingVar)); + } + sb.append(")"); + return sb.toString(); + } + + // toString(), trim() on safe navigation chain + if ("toString".equals(methodName) || "trim".equals(methodName)) { + final String obj = visitValueExpression(objExpr, bindingVar); + if (isSafe) { + return "(" + obj + " == null ? null : " + obj + "." + methodName + "())"; + } + return obj + "." + methodName + "()"; + } + + // Generic method call + final String obj = visitValueExpression(objExpr, bindingVar); + final StringBuilder sb = new StringBuilder(); + if (isSafe) { + sb.append("(").append(obj).append(" == null ? null : "); + } + sb.append(obj).append(".").append(methodName).append("("); + final List argExprs = args.getExpressions(); + for (int i = 0; i < argExprs.size(); i++) { + if (i > 0) { + sb.append(", "); + } + sb.append(visitValueExpression(argExprs.get(i), bindingVar)); + } + sb.append(")"); + if (isSafe) { + sb.append(")"); + } + return sb.toString(); + } + + private String visitGString(final GStringExpression expr, final String bindingVar) { + final List strings = expr.getStrings(); + final List values = expr.getValues(); + + final StringBuilder sb = new StringBuilder(); + for (int i = 0; i < strings.size(); i++) { + final String text = strings.get(i).getText(); + if (!text.isEmpty()) { + if (sb.length() > 0) { + sb.append(" + "); + } + sb.append("\"").append(MalToJavaTranspiler.escapeJava(text)).append("\""); + } + if (i < values.size()) { + final String val = visitValueExpression(values.get(i), bindingVar); + if (sb.length() > 0) { + sb.append(" + "); + } + sb.append(val); + } + } + return sb.length() > 0 ? sb.toString() : "\"\""; + } + + private String visitMapExpression(final MapExpression expr, final String bindingVar) { + final List entries = expr.getMapEntryExpressions(); + if (entries.isEmpty()) { + return "java.util.Collections.emptyMap()"; + } + if (entries.size() == 1) { + final MapEntryExpression e = entries.get(0); + return "Map.of(" + visitValueExpression(e.getKeyExpression(), bindingVar) + + ", " + visitValueExpression(e.getValueExpression(), bindingVar) + ")"; + } + final StringBuilder sb = new StringBuilder("Map.of("); + for (int i = 0; i < entries.size(); i++) { + if (i > 0) { + sb.append(", "); + } + final MapEntryExpression e = entries.get(i); + sb.append(visitValueExpression(e.getKeyExpression(), bindingVar)); + sb.append(", "); + sb.append(visitValueExpression(e.getValueExpression(), bindingVar)); + } + sb.append(")"); + return sb.toString(); + } + + // ---- Helpers ---- + + private List flattenStatements(final Statement stmt) { + if (stmt instanceof BlockStatement) { + return ((BlockStatement) stmt).getStatements(); + } + return List.of(stmt); + } + + private boolean isThisOrImplicit(final Expression expr) { + if (expr instanceof VariableExpression) { + final String name = ((VariableExpression) expr).getName(); + return "this".equals(name); + } + return false; + } + + private String lambdaVarFor(final String methodName) { + switch (methodName) { + case "extractor": return "ext"; + case "sink": return "s"; + case "slowSql": return "sql"; + case "sampledTrace": return "st"; + case "metrics": return "m"; + case "sampler": return "sp"; + case "text": return "tp"; + default: return "x"; + } + } + + private String childReceiverFor(final String lambdaVar) { + return lambdaVar; + } + + private ArgumentListExpression toArgList(final Expression args) { + if (args instanceof ArgumentListExpression) { + return (ArgumentListExpression) args; + } + if (args instanceof TupleExpression) { + final ArgumentListExpression ale = new ArgumentListExpression(); + for (Expression e : ((TupleExpression) args).getExpressions()) { + ale.addExpression(e); + } + return ale; + } + final ArgumentListExpression ale = new ArgumentListExpression(); + ale.addExpression(args); + return ale; + } + + private static String capitalize(final String s) { + if (s == null || s.isEmpty()) { + return s; + } + return Character.toUpperCase(s.charAt(0)) + s.substring(1); + } + + private static void indent(final StringBuilder sb, final int level) { + for (int i = 0; i < level; i++) { + sb.append(" "); + } + } + + // ---- Compilation & Manifest ---- + + /** + * Compile all registered LAL sources using javax.tools.JavaCompiler. + */ + public void compileAll(final File sourceDir, final File outputDir, + final String classpath) throws IOException { + if (lalSources.isEmpty()) { + log.info("No LAL sources to compile."); + return; + } + + final String packageDir = GENERATED_PACKAGE.replace('.', File.separatorChar); + final File srcPkgDir = new File(sourceDir, packageDir); + if (!srcPkgDir.exists() && !srcPkgDir.mkdirs()) { + throw new IOException("Failed to create source dir: " + srcPkgDir); + } + if (!outputDir.exists() && !outputDir.mkdirs()) { + throw new IOException("Failed to create output dir: " + outputDir); + } + + final List javaFiles = new ArrayList<>(); + for (Map.Entry entry : lalSources.entrySet()) { + final File javaFile = new File(srcPkgDir, entry.getKey() + ".java"); + Files.writeString(javaFile.toPath(), entry.getValue()); + javaFiles.add(javaFile); + } + + final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + if (compiler == null) { + throw new IllegalStateException("No Java compiler available — requires JDK"); + } + + final StringWriter errorWriter = new StringWriter(); + + try (StandardJavaFileManager fileManager = compiler.getStandardFileManager(null, null, null)) { + final Iterable compilationUnits = + fileManager.getJavaFileObjectsFromFiles(javaFiles); + + final List options = Arrays.asList( + "-d", outputDir.getAbsolutePath(), + "-classpath", classpath + ); + + final JavaCompiler.CompilationTask task = compiler.getTask( + errorWriter, fileManager, null, options, null, compilationUnits); + + final boolean success = task.call(); + if (!success) { + for (Map.Entry entry : lalSources.entrySet()) { + log.error("Generated source for {}:\n{}", entry.getKey(), entry.getValue()); + } + throw new RuntimeException( + "Java compilation failed for " + javaFiles.size() + " LAL sources:\n" + + errorWriter); + } + } + + log.info("Compiled {} LAL sources to {}", lalSources.size(), outputDir); + } + + /** + * Write lal-expressions.txt manifest: hash=FQCN format. + */ + public void writeManifest(final File outputDir) throws IOException { + final File manifestDir = new File(outputDir, "META-INF"); + if (!manifestDir.exists() && !manifestDir.mkdirs()) { + throw new IOException("Failed to create META-INF dir: " + manifestDir); + } + + final List lines = hashToClass.entrySet().stream() + .map(e -> e.getKey() + "=" + e.getValue()) + .sorted() + .collect(Collectors.toList()); + Files.write(new File(manifestDir, "lal-expressions.txt").toPath(), lines); + log.info("Wrote lal-expressions.txt with {} entries", lines.size()); + } +} diff --git a/oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java b/oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java new file mode 100644 index 000000000000..4a580a4bcd36 --- /dev/null +++ b/oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java @@ -0,0 +1,678 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.transpiler.lal; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class LalToJavaTranspilerTest { + + private LalToJavaTranspiler transpiler; + + @BeforeEach + void setUp() { + transpiler = new LalToJavaTranspiler(); + } + + // ---- Class Structure ---- + + @Test + void classStructure() { + final String java = transpiler.transpile("LalExpr_test", + "filter { json {} }"); + assertNotNull(java); + + assertTrue(java.contains("package " + LalToJavaTranspiler.GENERATED_PACKAGE), + "Should have correct package"); + assertTrue(java.contains("public class LalExpr_test implements LalExpression"), + "Should implement LalExpression"); + assertTrue(java.contains("public void execute(FilterSpec filterSpec, Binding binding)"), + "Should have execute method"); + } + + @Test + void helperMethodsPresent() { + final String java = transpiler.transpile("LalExpr_test", + "filter { json {} }"); + assertNotNull(java); + + assertTrue(java.contains("private static Object getAt(Object obj, String key)"), + "Should have getAt helper"); + assertTrue(java.contains("private static long toLong(Object obj)"), + "Should have toLong helper"); + assertTrue(java.contains("private static int toInt(Object obj)"), + "Should have toInt helper"); + assertTrue(java.contains("private static boolean toBoolean(Object obj)"), + "Should have toBoolean helper"); + assertTrue(java.contains("private static boolean isTruthy(Object obj)"), + "Should have isTruthy helper"); + assertTrue(java.contains("private static boolean isNonEmptyString(Object obj)"), + "Should have isNonEmptyString helper"); + } + + @Test + void importsPresent() { + final String java = transpiler.transpile("LalExpr_test", + "filter { json {} }"); + assertNotNull(java); + + assertTrue(java.contains("import org.apache.skywalking.oap.log.analyzer.dsl.Binding;"), + "Should import Binding"); + assertTrue(java.contains("import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression;"), + "Should import LalExpression"); + assertTrue(java.contains("import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec;"), + "Should import FilterSpec"); + } + + // ---- filter {} Unwrapping ---- + + @Test + void filterUnwrap() { + final String java = transpiler.transpile("LalExpr_test", + "filter { json {} }"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.json()"), + "Should unwrap filter block and call json() on filterSpec"); + assertTrue(!java.contains("filterSpec.filter("), + "Should NOT emit filterSpec.filter() call"); + } + + // ---- json {} ---- + + @Test + void jsonEmptyBlock() { + final String java = transpiler.transpile("LalExpr_test", + "filter { json {} }"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.json();"), + "Should emit no-arg json() call"); + } + + // ---- text { regexp $/pattern/$ } ---- + + @Test + void textWithRegexp() { + final String java = transpiler.transpile("LalExpr_test", + "filter { text { regexp $/(?\\d+)\\s+(?.+)/$ } }"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.text(tp -> {"), + "Should emit text with Consumer lambda"); + assertTrue(java.contains("tp.regexp("), + "Should call regexp on text parser spec"); + } + + // ---- extractor { ... } ---- + + @Test + void extractorWithService() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed.service as String\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.extractor(ext -> {"), + "Should emit extractor with Consumer lambda"); + assertTrue(java.contains("ext.service("), + "Should call service on extractor spec"); + assertTrue(java.contains("String.valueOf("), + "Should use String.valueOf for 'as String' cast"); + assertTrue(java.contains("getAt(binding.parsed(), \"service\")"), + "Should use getAt for parsed.service"); + } + + @Test + void extractorWithMultipleFields() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed.service as String\n" + + " instance parsed.instance as String\n" + + " endpoint parsed.endpoint as String\n" + + " layer parsed.layer as String\n" + + " timestamp parsed.time as String\n" + + " traceId parsed.traceId as String\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("ext.service(String.valueOf(getAt(binding.parsed(), \"service\")))"), + "Should extract service"); + assertTrue(java.contains("ext.instance(String.valueOf(getAt(binding.parsed(), \"instance\")))"), + "Should extract instance"); + assertTrue(java.contains("ext.endpoint(String.valueOf(getAt(binding.parsed(), \"endpoint\")))"), + "Should extract endpoint"); + assertTrue(java.contains("ext.layer(String.valueOf(getAt(binding.parsed(), \"layer\")))"), + "Should extract layer"); + assertTrue(java.contains("ext.timestamp(String.valueOf(getAt(binding.parsed(), \"time\")))"), + "Should extract timestamp"); + assertTrue(java.contains("ext.traceId(String.valueOf(getAt(binding.parsed(), \"traceId\")))"), + "Should extract traceId"); + } + + // ---- sink { ... } ---- + + @Test + void sinkWithEnforcer() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " sink {\n" + + " enforcer {}\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.sink(s -> {"), + "Should emit sink with Consumer lambda"); + assertTrue(java.contains("s.enforcer();"), + "Should emit no-arg enforcer() on sink spec"); + } + + @Test + void sinkWithDropper() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " sink {\n" + + " dropper {}\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("s.dropper();"), + "Should emit no-arg dropper() on sink spec"); + } + + @Test + void sinkWithSampler() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit('abc')\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("s.sampler(sp -> {"), + "Should emit sampler with Consumer lambda"); + assertTrue(java.contains("sp.rateLimit(\"abc\")"), + "Should call rateLimit on sampler spec"); + } + + // ---- abort {} ---- + + @Test + void abortBlock() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " abort {}\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.abort();"), + "Should emit no-arg abort() call"); + } + + // ---- parsed access ---- + + @Test + void parsedPropertyAccess() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { service parsed.service as String }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("binding.parsed()"), + "Should translate 'parsed' to binding.parsed()"); + assertTrue(java.contains("getAt(binding.parsed(), \"service\")"), + "Should use getAt for property access"); + } + + @Test + void parsedNestedAccess() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { service parsed.data.serviceName as String }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("getAt(getAt(binding.parsed(), \"data\"), \"serviceName\")"), + "Should translate nested parsed access to nested getAt()"); + } + + // ---- Safe navigation (?.) ---- + + @Test + void safeNavigationOnParsed() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { service parsed?.service as String }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("== null ? null : getAt("), + "Should translate ?. to null-safe ternary with getAt"); + } + + // ---- as Cast ---- + + @Test + void castAsString() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { service parsed.service as String }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("String.valueOf("), + "Should translate 'as String' to String.valueOf()"); + } + + @Test + void castAsLong() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { timestamp parsed.time as Long }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("toLong("), + "Should translate 'as Long' to toLong()"); + } + + // ---- log access ---- + + @Test + void logPropertyAccess() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { service log.service }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("binding.log().getService()"), + "Should translate log.service to binding.log().getService()"); + } + + // ---- if/else ---- + + @Test + void ifStatement() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.level == 'ERROR') {\n" + + " abort {}\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("if (\"ERROR\".equals(getAt(binding.parsed(), \"level\"))"), + "Should translate == with constant on left for null-safety"); + assertTrue(java.contains("filterSpec.abort()"), + "Should emit abort in if body"); + } + + @Test + void ifElseStatement() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.type == 'access') {\n" + + " extractor { layer 'HTTP' }\n" + + " } else {\n" + + " extractor { layer 'GENERAL' }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("if (\"access\".equals(getAt(binding.parsed(), \"type\"))"), + "Should have if condition"); + assertTrue(java.contains("} else {"), + "Should have else block"); + } + + @Test + void ifElseIfStatement() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.type == 'a') {\n" + + " extractor { layer 'A' }\n" + + " } else if (parsed.type == 'b') {\n" + + " extractor { layer 'B' }\n" + + " } else {\n" + + " extractor { layer 'C' }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("} else if ("), + "Should produce else-if chain"); + } + + // ---- Condition operators ---- + + @Test + void notEqualCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.status != 'ok') { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("!\"ok\".equals(getAt(binding.parsed(), \"status\"))"), + "Should translate != with negated .equals()"); + } + + @Test + void logicalAndCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.a == 'x' && parsed.b == 'y') { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("\"x\".equals(getAt(binding.parsed(), \"a\")) && \"y\".equals(getAt(binding.parsed(), \"b\"))"), + "Should translate && with .equals() on both sides"); + } + + @Test + void logicalOrCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.a == 'x' || parsed.a == 'y') { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("\"x\".equals(") && java.contains("|| \"y\".equals("), + "Should translate || correctly"); + } + + @Test + void truthinessCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.value) { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("isTruthy(getAt(binding.parsed(), \"value\"))"), + "Should translate bare expression to isTruthy()"); + } + + @Test + void negationCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (!parsed.value) { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("!isTruthy(getAt(binding.parsed(), \"value\"))"), + "Should translate !expr to negated isTruthy()"); + } + + @Test + void lessThanCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.code < 400) { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("toInt(getAt(binding.parsed(), \"code\")) < 400"), + "Should translate < with toInt on left"); + } + + @Test + void greaterThanEqualCondition() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (parsed.code >= 500) { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("toInt(getAt(binding.parsed(), \"code\")) >= 500"), + "Should translate >= with toInt on left"); + } + + // ---- GString interpolation ---- + + @Test + void gstringInterpolation() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor { service \"svc::${parsed.name}\" as String }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("\"svc::\""), + "Should have string prefix"); + assertTrue(java.contains("getAt(binding.parsed(), \"name\")"), + "Should have parsed access in interpolation"); + } + + // ---- tag(Map) ---- + + @Test + void tagWithMapLiteral() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " tag(status: parsed.status as String)\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("ext.tag("), + "Should call tag on extractor spec"); + } + + // ---- slowSql / sampledTrace / metrics ---- + + @Test + void slowSqlBlock() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " layer 'MYSQL'\n" + + " service parsed.service as String\n" + + " slowSql {\n" + + " id parsed.id as String\n" + + " statement parsed.statement as String\n" + + " latency parsed.latency as Long\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("ext.slowSql(sql -> {"), + "Should emit slowSql with Consumer lambda, var 'sql'"); + assertTrue(java.contains("sql.id("), + "Should call id on slowSql spec"); + assertTrue(java.contains("sql.statement("), + "Should call statement on slowSql spec"); + assertTrue(java.contains("sql.latency(toLong("), + "Should call latency with toLong"); + } + + @Test + void sampledTraceBlock() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " sampledTrace {\n" + + " uri parsed.uri as String\n" + + " latency parsed.latency as Long\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("ext.sampledTrace(st -> {"), + "Should emit sampledTrace with Consumer lambda, var 'st'"); + assertTrue(java.contains("st.uri("), + "Should call uri on sampledTrace spec"); + } + + @Test + void metricsBlock() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " metrics {\n" + + " name 'log_count'\n" + + " value 1\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("ext.metrics(m -> {"), + "Should emit metrics with Consumer lambda, var 'm'"); + assertTrue(java.contains("m.name(\"log_count\")"), + "Should call name on metrics spec"); + assertTrue(java.contains("m.value(1)"), + "Should call value on metrics spec"); + } + + // ---- Complete LAL Script ---- + + @Test + void completeScript() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed.service as String\n" + + " instance parsed.instance as String\n" + + " layer parsed.layer as String\n" + + " timestamp parsed.time as String\n" + + " }\n" + + " sink {\n" + + " enforcer {}\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.json();"), + "Should have json() call"); + assertTrue(java.contains("filterSpec.extractor(ext -> {"), + "Should have extractor block"); + assertTrue(java.contains("filterSpec.sink(s -> {"), + "Should have sink block"); + assertTrue(java.contains("s.enforcer();"), + "Should have enforcer in sink"); + } + + // ---- tag("KEY") as value ---- + + @Test + void tagAsValue() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " if (tag('status') == 'error') { abort {} }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("filterSpec.tag(\"status\")"), + "Should call tag on filterSpec"); + } + + // ---- rateLimit ---- + + @Test + void rateLimitWithClosureArg() { + final String java = transpiler.transpile("LalExpr_test", + "filter {\n" + + " json {}\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit('myId') { rpm 5 }\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(java); + + assertTrue(java.contains("sp.rateLimit(\"myId\", rls -> {"), + "Should emit rateLimit with id and closure lambda"); + assertTrue(java.contains("rls.rpm(5)"), + "Should call rpm on rate limit spec"); + } + + // ---- Manifest ---- + + @Test + void manifest(@TempDir Path tempDir) throws Exception { + final String source = transpiler.transpile("LalExpr_a", "filter { json {} }"); + transpiler.register("LalExpr_a", "abc123hash", source); + + final File outputDir = tempDir.toFile(); + transpiler.writeManifest(outputDir); + + final File manifest = new File(outputDir, "META-INF/lal-expressions.txt"); + assertTrue(manifest.exists(), "Manifest file should exist"); + + final List lines = Files.readAllLines(manifest.toPath()); + assertTrue(lines.stream().anyMatch(l -> l.contains("abc123hash") && + l.contains(LalToJavaTranspiler.GENERATED_PACKAGE + ".LalExpr_a")), + "Should contain hash=FQCN mapping"); + } +} diff --git a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java index c256bb589cb6..d7632688e17a 100644 --- a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java +++ b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java @@ -1089,7 +1089,7 @@ private List visitArgList(final ArgumentListExpression args) { // ---- Utility ---- - static String escapeJava(final String s) { + public static String escapeJava(final String s) { return s.replace("\\", "\\\\") .replace("\"", "\\\"") .replace("\n", "\\n") diff --git a/oap-server/analyzer/pom.xml b/oap-server/analyzer/pom.xml index 4039928a5023..1c2a04f34dc4 100644 --- a/oap-server/analyzer/pom.xml +++ b/oap-server/analyzer/pom.xml @@ -34,6 +34,7 @@ meter-analyzer event-analyzer mal-transpiler + lal-transpiler From e9a6c9d54984f0577e9211ad12e8bb7fd165942e Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 15:58:56 +0800 Subject: [PATCH 05/64] Add v2 runtime modules for MAL and LAL with manifest-based class loading (Phase 4) Introduces meter-analyzer-v2 and log-analyzer-v2 modules that provide same-FQCN replacement classes for DSL.java, Expression.java, and FilterExpression.java. The v2 classes load transpiled MalExpression/ MalFilter/LalExpression implementations from META-INF manifests via Class.forName() instead of Groovy GroovyShell/ExpandoMetaClass/ DelegatingScript. Uses maven-shade-plugin to overlay the upstream Groovy-dependent classes. Includes 7 unit tests. Co-Authored-By: Claude Opus 4.6 --- oap-server/analyzer/log-analyzer-v2/pom.xml | 72 +++++++++ .../skywalking/oap/log/analyzer/dsl/DSL.java | 142 ++++++++++++++++++ .../oap/log/analyzer/dsl/DSLV2Test.java | 60 ++++++++ oap-server/analyzer/meter-analyzer-v2/pom.xml | 78 ++++++++++ .../oap/meter/analyzer/dsl/DSL.java | 116 ++++++++++++++ .../oap/meter/analyzer/dsl/Expression.java | 92 ++++++++++++ .../meter/analyzer/dsl/FilterExpression.java | 113 ++++++++++++++ .../oap/meter/analyzer/dsl/DSLV2Test.java | 98 ++++++++++++ oap-server/analyzer/pom.xml | 2 + 9 files changed, 773 insertions(+) create mode 100644 oap-server/analyzer/log-analyzer-v2/pom.xml create mode 100644 oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java create mode 100644 oap-server/analyzer/log-analyzer-v2/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java create mode 100644 oap-server/analyzer/meter-analyzer-v2/pom.xml create mode 100644 oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java create mode 100644 oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java create mode 100644 oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java create mode 100644 oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java diff --git a/oap-server/analyzer/log-analyzer-v2/pom.xml b/oap-server/analyzer/log-analyzer-v2/pom.xml new file mode 100644 index 000000000000..6456f8209f16 --- /dev/null +++ b/oap-server/analyzer/log-analyzer-v2/pom.xml @@ -0,0 +1,72 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + log-analyzer-v2 + Pure Java LAL runtime that loads transpiled LalExpression classes from manifest instead of Groovy + + + + org.apache.skywalking + log-analyzer + ${project.version} + + + + + + + org.apache.maven.plugins + maven-shade-plugin + + + package + + shade + + + true + + + org.apache.skywalking:log-analyzer + + + + + org.apache.skywalking:log-analyzer + + org/apache/skywalking/oap/log/analyzer/dsl/DSL.class + org/apache/skywalking/oap/log/analyzer/dsl/DSL$*.class + + + + + + + + + + diff --git a/oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java b/oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java new file mode 100644 index 000000000000..b3f4e4c977d2 --- /dev/null +++ b/oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java @@ -0,0 +1,142 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.dsl; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import lombok.AccessLevel; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; + +/** + * Same-FQCN replacement for upstream LAL DSL. + * Loads pre-compiled {@link LalExpression} classes from lal-expressions.txt manifest + * (keyed by SHA-256 hash) instead of Groovy {@code GroovyShell} runtime compilation. + */ +@Slf4j +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +public class DSL { + private static final String MANIFEST_PATH = "META-INF/lal-expressions.txt"; + private static volatile Map EXPRESSION_MAP; + private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + + private final LalExpression expression; + private final FilterSpec filterSpec; + private Binding binding; + + public static DSL of(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig config, + final String dsl) throws ModuleStartException { + final Map exprMap = loadManifest(); + final String dslHash = sha256(dsl); + final String className = exprMap.get(dslHash); + if (className == null) { + throw new ModuleStartException( + "Pre-compiled LAL expression not found for DSL hash: " + dslHash + + ". Available: " + exprMap.size() + " expressions."); + } + + try { + final Class exprClass = Class.forName(className); + final LalExpression expression = (LalExpression) exprClass.getDeclaredConstructor().newInstance(); + final FilterSpec filterSpec = new FilterSpec(moduleManager, config); + final int count = LOADED_COUNT.incrementAndGet(); + log.debug("Loaded pre-compiled LAL expression [{}/{}]: {}", count, exprMap.size(), className); + return new DSL(expression, filterSpec); + } catch (ClassNotFoundException e) { + throw new ModuleStartException( + "Pre-compiled LAL expression class not found: " + className, e); + } catch (ReflectiveOperationException e) { + throw new ModuleStartException( + "Pre-compiled LAL expression instantiation failed: " + className, e); + } + } + + public void bind(final Binding binding) { + this.binding = binding; + this.filterSpec.bind(binding); + } + + public void evaluate() { + expression.execute(filterSpec, binding); + } + + private static Map loadManifest() { + if (EXPRESSION_MAP != null) { + return EXPRESSION_MAP; + } + synchronized (DSL.class) { + if (EXPRESSION_MAP != null) { + return EXPRESSION_MAP; + } + final Map map = new HashMap<>(); + try (InputStream is = DSL.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { + if (is == null) { + log.warn("LAL expression manifest not found: {}", MANIFEST_PATH); + EXPRESSION_MAP = map; + return map; + } + try (BufferedReader reader = new BufferedReader( + new InputStreamReader(is, StandardCharsets.UTF_8))) { + String line; + while ((line = reader.readLine()) != null) { + line = line.trim(); + if (line.isEmpty()) { + continue; + } + final String[] parts = line.split("=", 2); + if (parts.length == 2) { + map.put(parts[0], parts[1]); + } + } + } + } catch (IOException e) { + throw new IllegalStateException("Failed to load LAL expression manifest", e); + } + log.info("Loaded {} pre-compiled LAL expressions from manifest", map.size()); + EXPRESSION_MAP = map; + return map; + } + } + + static String sha256(final String input) { + try { + final MessageDigest digest = MessageDigest.getInstance("SHA-256"); + final byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8)); + final StringBuilder hex = new StringBuilder(); + for (final byte b : hash) { + hex.append(String.format("%02x", b)); + } + return hex.toString(); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException("SHA-256 not available", e); + } + } +} diff --git a/oap-server/analyzer/log-analyzer-v2/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java b/oap-server/analyzer/log-analyzer-v2/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java new file mode 100644 index 000000000000..a8f19d382e41 --- /dev/null +++ b/oap-server/analyzer/log-analyzer-v2/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.dsl; + +import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class DSLV2Test { + + @Test + void ofThrowsWhenManifestMissing() { + // No META-INF/lal-expressions.txt on test classpath + assertThrows(ModuleStartException.class, + () -> DSL.of(null, null, "filter { json {} sink {} }")); + } + + @Test + void sha256Deterministic() { + final String input = "filter { json {} sink {} }"; + final String hash1 = DSL.sha256(input); + final String hash2 = DSL.sha256(input); + assertNotNull(hash1); + assertEquals(64, hash1.length()); + assertEquals(hash1, hash2); + } + + @Test + void sha256DifferentInputsDifferentHashes() { + final String hash1 = DSL.sha256("filter { json {} sink {} }"); + final String hash2 = DSL.sha256("filter { text {} sink {} }"); + assertNotNull(hash1); + assertNotNull(hash2); + assertNotEquals(hash1, hash2); + } + + private static void assertNotEquals(final String a, final String b) { + if (a.equals(b)) { + throw new AssertionError("Expected different values but got: " + a); + } + } +} diff --git a/oap-server/analyzer/meter-analyzer-v2/pom.xml b/oap-server/analyzer/meter-analyzer-v2/pom.xml new file mode 100644 index 000000000000..8bbcaa43ed47 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer-v2/pom.xml @@ -0,0 +1,78 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + meter-analyzer-v2 + Pure Java MAL runtime that loads transpiled MalExpression/MalFilter classes from manifests instead of Groovy + + + + org.apache.skywalking + meter-analyzer + ${project.version} + + + + + + + org.apache.maven.plugins + maven-shade-plugin + + + package + + shade + + + true + + + org.apache.skywalking:meter-analyzer + + + + + org.apache.skywalking:meter-analyzer + + org/apache/skywalking/oap/meter/analyzer/dsl/DSL.class + org/apache/skywalking/oap/meter/analyzer/dsl/DSL$*.class + org/apache/skywalking/oap/meter/analyzer/dsl/Expression.class + org/apache/skywalking/oap/meter/analyzer/dsl/Expression$*.class + org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.class + org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression$*.class + org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.class + org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure$*.class + + + + + + + + + + diff --git a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java new file mode 100644 index 000000000000..3098ac7672d7 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import lombok.extern.slf4j.Slf4j; + +/** + * Same-FQCN replacement for upstream MAL DSL. + * Loads transpiled {@link MalExpression} classes from mal-expressions.txt manifest + * instead of Groovy {@code DelegatingScript} classes -- no Groovy runtime needed. + */ +@Slf4j +public final class DSL { + private static final String MANIFEST_PATH = "META-INF/mal-expressions.txt"; + private static volatile Map SCRIPT_MAP; + private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + + /** + * Parse string literal to Expression object, which can be reused. + * + * @param metricName the name of metric defined in mal rule + * @param expression string literal represents the DSL expression. + * @return Expression object could be executed. + */ + public static Expression parse(final String metricName, final String expression) { + if (metricName == null) { + throw new UnsupportedOperationException( + "Init expressions (metricName=null) are not supported in v2 mode. " + + "All init expressions must be pre-compiled at build time."); + } + + final Map scriptMap = loadManifest(); + final String className = scriptMap.get(metricName); + if (className == null) { + throw new IllegalStateException( + "Transpiled MAL expression not found for metric: " + metricName + + ". Available: " + scriptMap.size() + " expressions"); + } + + try { + final Class exprClass = Class.forName(className); + final MalExpression malExpr = (MalExpression) exprClass.getDeclaredConstructor().newInstance(); + final int count = LOADED_COUNT.incrementAndGet(); + log.debug("Loaded transpiled MAL expression [{}/{}]: {}", count, scriptMap.size(), metricName); + return new Expression(metricName, expression, malExpr); + } catch (ClassNotFoundException e) { + throw new IllegalStateException( + "Transpiled MAL expression class not found: " + className, e); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException( + "Failed to instantiate transpiled MAL expression: " + className, e); + } + } + + private static Map loadManifest() { + if (SCRIPT_MAP != null) { + return SCRIPT_MAP; + } + synchronized (DSL.class) { + if (SCRIPT_MAP != null) { + return SCRIPT_MAP; + } + final Map map = new HashMap<>(); + try (InputStream is = DSL.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { + if (is == null) { + log.warn("MAL expression manifest not found: {}", MANIFEST_PATH); + SCRIPT_MAP = map; + return map; + } + try (BufferedReader reader = new BufferedReader( + new InputStreamReader(is, StandardCharsets.UTF_8))) { + String line; + while ((line = reader.readLine()) != null) { + line = line.trim(); + if (line.isEmpty()) { + continue; + } + final String simpleName = line.substring(line.lastIndexOf('.') + 1); + if (simpleName.startsWith("MalExpr_")) { + final String metric = simpleName.substring("MalExpr_".length()); + map.put(metric, line); + } + } + } + } catch (IOException e) { + throw new IllegalStateException("Failed to load MAL expression manifest", e); + } + log.info("Loaded {} transpiled MAL expressions from manifest", map.size()); + SCRIPT_MAP = map; + return map; + } + } +} diff --git a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java new file mode 100644 index 000000000000..cf2e2083017c --- /dev/null +++ b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import com.google.common.collect.ImmutableMap; +import java.util.Map; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; + +/** + * Same-FQCN replacement for upstream Expression. + * Wraps a transpiled {@link MalExpression} (pure Java) instead of a Groovy DelegatingScript. + * No ExpandoMetaClass, no propertyMissing(), no ThreadLocal sample repository. + */ +@Slf4j +@ToString(of = {"literal"}) +public class Expression { + + private final String metricName; + private final String literal; + private final MalExpression expression; + + public Expression(final String metricName, final String literal, final MalExpression expression) { + this.metricName = metricName; + this.literal = literal; + this.expression = expression; + } + + /** + * Parse the expression statically. + * + * @return Parsed context of the expression. + */ + public ExpressionParsingContext parse() { + try (ExpressionParsingContext ctx = ExpressionParsingContext.create()) { + final Result r = run(ImmutableMap.of()); + if (!r.isSuccess() && r.isThrowable()) { + throw new ExpressionParsingException( + "failed to parse expression: " + literal + ", error:" + r.getError()); + } + if (log.isDebugEnabled()) { + log.debug("\"{}\" is parsed", literal); + } + ctx.validate(literal); + return ctx; + } + } + + /** + * Run the expression with a data map. + * + * @param sampleFamilies a data map includes all of candidates to be analysis. + * @return The result of execution. + */ + public Result run(final Map sampleFamilies) { + try { + for (final SampleFamily s : sampleFamilies.values()) { + if (s != SampleFamily.EMPTY) { + s.context.setMetricName(metricName); + } + } + final SampleFamily sf = expression.run(sampleFamilies); + if (sf == SampleFamily.EMPTY) { + if (ExpressionParsingContext.get().isEmpty()) { + if (log.isDebugEnabled()) { + log.debug("result of {} is empty by \"{}\"", sampleFamilies, literal); + } + } + return Result.fail("Parsed result is an EMPTY sample family"); + } + return Result.success(sf); + } catch (Throwable t) { + log.error("failed to run \"{}\"", literal, t); + return Result.fail(t); + } + } +} diff --git a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java new file mode 100644 index 000000000000..b92318d9d372 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java @@ -0,0 +1,113 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.io.IOException; +import java.io.InputStream; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; +import java.util.Properties; +import java.util.concurrent.atomic.AtomicInteger; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; + +/** + * Same-FQCN replacement for upstream FilterExpression. + * Loads transpiled {@link MalFilter} classes from mal-filter-expressions.properties + * manifest instead of Groovy filter closures -- no Groovy runtime needed. + */ +@Slf4j +@ToString(of = {"literal"}) +public class FilterExpression { + private static final String MANIFEST_PATH = "META-INF/mal-filter-expressions.properties"; + private static volatile Map FILTER_MAP; + private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + + private final String literal; + private final MalFilter malFilter; + + @SuppressWarnings("unchecked") + public FilterExpression(final String literal) { + this.literal = literal; + + final Map filterMap = loadManifest(); + final String className = filterMap.get(literal); + if (className == null) { + throw new IllegalStateException( + "Transpiled MAL filter not found for: " + literal + + ". Available filters: " + filterMap.size()); + } + + try { + final Class filterClass = Class.forName(className); + malFilter = (MalFilter) filterClass.getDeclaredConstructor().newInstance(); + final int count = LOADED_COUNT.incrementAndGet(); + log.debug("Loaded transpiled MAL filter [{}/{}]: {}", count, filterMap.size(), literal); + } catch (ClassNotFoundException e) { + throw new IllegalStateException( + "Transpiled MAL filter class not found: " + className, e); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException( + "Failed to instantiate transpiled MAL filter: " + className, e); + } + } + + public Map filter(final Map sampleFamilies) { + try { + final Map result = new HashMap<>(); + for (final Map.Entry entry : sampleFamilies.entrySet()) { + final SampleFamily afterFilter = entry.getValue().filter(malFilter::test); + if (!Objects.equals(afterFilter, SampleFamily.EMPTY)) { + result.put(entry.getKey(), afterFilter); + } + } + return result; + } catch (Throwable t) { + log.error("failed to run \"{}\"", literal, t); + } + return sampleFamilies; + } + + private static Map loadManifest() { + if (FILTER_MAP != null) { + return FILTER_MAP; + } + synchronized (FilterExpression.class) { + if (FILTER_MAP != null) { + return FILTER_MAP; + } + final Map map = new HashMap<>(); + try (InputStream is = FilterExpression.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { + if (is == null) { + log.warn("MAL filter manifest not found: {}", MANIFEST_PATH); + FILTER_MAP = map; + return map; + } + final Properties props = new Properties(); + props.load(is); + props.forEach((k, v) -> map.put((String) k, (String) v)); + } catch (IOException e) { + throw new IllegalStateException("Failed to load MAL filter manifest", e); + } + log.info("Loaded {} transpiled MAL filters from manifest", map.size()); + FILTER_MAP = map; + return map; + } + } +} diff --git a/oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java b/oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java new file mode 100644 index 000000000000..d64847e77186 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import com.google.common.collect.ImmutableMap; +import java.util.HashMap; +import java.util.Map; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DSLV2Test { + + @Test + void parseRejectsNullMetricName() { + assertThrows(UnsupportedOperationException.class, () -> DSL.parse(null, "test_metric")); + } + + @Test + void parseThrowsWhenManifestMissing() { + assertThrows(IllegalStateException.class, () -> DSL.parse("nonexistent_metric", "some_expr")); + } + + @Test + void expressionRunWithMalExpression() { + final MalExpression simple = samples -> + samples.getOrDefault("test_metric", SampleFamily.EMPTY); + + final Expression expr = new Expression("test_metric", "test_metric", simple); + + // Run with empty map should return fail (EMPTY) + final Result emptyResult = expr.run(Map.of()); + assertNotNull(emptyResult); + assertFalse(emptyResult.isSuccess()); + + // Run with a real sample should return success + final Sample sample = Sample.builder() + .name("test_metric") + .labels(ImmutableMap.of("service", "svc1")) + .value(42.0) + .timestamp(System.currentTimeMillis()) + .build(); + final SampleFamily sf = SampleFamily.build(SampleFamily.RunningContext.instance(), sample); + final Map sampleMap = new HashMap<>(); + sampleMap.put("test_metric", sf); + + final Result result = expr.run(sampleMap); + assertNotNull(result); + assertTrue(result.isSuccess()); + assertEquals(sf, result.getData()); + } + + @Test + void filterExpressionWithMalFilter() { + final MalFilter filter = tags -> "svc1".equals(tags.get("service")); + + final Sample sample1 = Sample.builder() + .name("metric") + .labels(ImmutableMap.of("service", "svc1")) + .value(10.0) + .timestamp(System.currentTimeMillis()) + .build(); + final Sample sample2 = Sample.builder() + .name("metric") + .labels(ImmutableMap.of("service", "svc2")) + .value(20.0) + .timestamp(System.currentTimeMillis()) + .build(); + + final SampleFamily sf = SampleFamily.build( + SampleFamily.RunningContext.instance(), sample1, sample2); + + final SampleFamily filtered = sf.filter(filter::test); + assertNotNull(filtered); + assertTrue(filtered != SampleFamily.EMPTY); + assertEquals(1, filtered.samples.length); + assertEquals(10.0, filtered.samples[0].getValue()); + } +} diff --git a/oap-server/analyzer/pom.xml b/oap-server/analyzer/pom.xml index 1c2a04f34dc4..5053d0e62858 100644 --- a/oap-server/analyzer/pom.xml +++ b/oap-server/analyzer/pom.xml @@ -35,6 +35,8 @@ event-analyzer mal-transpiler lal-transpiler + meter-analyzer-v2 + log-analyzer-v2 From 0b8ec1cdb5cba3443e5e72c729a332fc6ffbf987 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 16:13:57 +0800 Subject: [PATCH 06/64] Refactor hierarchy rules: replace Groovy Closure with Java BiFunction (Phase 5) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extract hierarchy matching rules from HierarchyDefinitionService into pluggable HierarchyRuleProvider interface. Remove Groovy imports from server-core by replacing Closure with BiFunction. - hierarchy-v1: GroovyHierarchyRuleProvider (for CI checker only) - hierarchy-v2: JavaHierarchyRuleProvider with 4 built-in rules + 12 tests - HierarchyDefinitionService: add HierarchyRuleProvider interface, DefaultJavaRuleProvider - HierarchyService: .getClosure().call() → .match() Co-Authored-By: Claude Opus 4.6 --- oap-server/analyzer/hierarchy-v1/pom.xml | 42 +++++ .../config/GroovyHierarchyRuleProvider.java | 49 ++++++ oap-server/analyzer/hierarchy-v2/pom.xml | 38 +++++ .../config/JavaHierarchyRuleProvider.java | 86 ++++++++++ .../config/JavaHierarchyRuleProviderTest.java | 155 ++++++++++++++++++ oap-server/analyzer/pom.xml | 2 + .../config/HierarchyDefinitionService.java | 109 +++++++++--- .../core/hierarchy/HierarchyService.java | 10 +- 8 files changed, 464 insertions(+), 27 deletions(-) create mode 100644 oap-server/analyzer/hierarchy-v1/pom.xml create mode 100644 oap-server/analyzer/hierarchy-v1/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java create mode 100644 oap-server/analyzer/hierarchy-v2/pom.xml create mode 100644 oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java create mode 100644 oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java diff --git a/oap-server/analyzer/hierarchy-v1/pom.xml b/oap-server/analyzer/hierarchy-v1/pom.xml new file mode 100644 index 000000000000..0e2cf572f295 --- /dev/null +++ b/oap-server/analyzer/hierarchy-v1/pom.xml @@ -0,0 +1,42 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + hierarchy-v1 + Groovy-based hierarchy rule provider (for checker module only, not runtime) + + + + org.apache.skywalking + server-core + ${project.version} + + + org.apache.groovy + groovy + + + diff --git a/oap-server/analyzer/hierarchy-v1/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java b/oap-server/analyzer/hierarchy-v1/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java new file mode 100644 index 000000000000..c8e8af43056c --- /dev/null +++ b/oap-server/analyzer/hierarchy-v1/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config; + +import groovy.lang.Closure; +import groovy.lang.GroovyShell; +import java.util.HashMap; +import java.util.Map; +import java.util.function.BiFunction; +import org.apache.skywalking.oap.server.core.query.type.Service; + +/** + * Groovy-based hierarchy rule provider. Uses GroovyShell.evaluate() to compile + * hierarchy matching rule closures from YAML expressions. + * + *

This provider is NOT included in the runtime classpath. It is only used + * by the hierarchy-v1-v2-checker module for CI validation against the pure Java + * provider (hierarchy-v2). + */ +public final class GroovyHierarchyRuleProvider implements HierarchyDefinitionService.HierarchyRuleProvider { + + @Override + @SuppressWarnings("unchecked") + public Map> buildRules( + final Map ruleExpressions) { + final Map> rules = new HashMap<>(); + final GroovyShell sh = new GroovyShell(); + ruleExpressions.forEach((name, expression) -> { + final Closure closure = (Closure) sh.evaluate(expression); + rules.put(name, (u, l) -> closure.call(u, l)); + }); + return rules; + } +} diff --git a/oap-server/analyzer/hierarchy-v2/pom.xml b/oap-server/analyzer/hierarchy-v2/pom.xml new file mode 100644 index 000000000000..42e49678b2e6 --- /dev/null +++ b/oap-server/analyzer/hierarchy-v2/pom.xml @@ -0,0 +1,38 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + hierarchy-v2 + Pure Java hierarchy rule provider with static rule registry (no Groovy) + + + + org.apache.skywalking + server-core + ${project.version} + + + diff --git a/oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java b/oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java new file mode 100644 index 000000000000..d85bdb99c766 --- /dev/null +++ b/oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config; + +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; +import java.util.function.BiFunction; +import org.apache.skywalking.oap.server.core.query.type.Service; + +/** + * Pure Java hierarchy rule provider. Contains a static registry of all known + * hierarchy matching rules as Java lambdas. Zero Groovy dependency. + * + *

Rule names must match those in hierarchy-definition.yml auto-matching-rules section. + * Unknown rule names fail fast at startup with IllegalArgumentException. + */ +public final class JavaHierarchyRuleProvider implements HierarchyDefinitionService.HierarchyRuleProvider { + + private static final Map> RULE_REGISTRY; + + static { + RULE_REGISTRY = new HashMap<>(); + + // name: { (u, l) -> u.name == l.name } + RULE_REGISTRY.put("name", + (u, l) -> Objects.equals(u.getName(), l.getName())); + + // short-name: { (u, l) -> u.shortName == l.shortName } + RULE_REGISTRY.put("short-name", + (u, l) -> Objects.equals(u.getShortName(), l.getShortName())); + + // lower-short-name-remove-ns: + // { (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) + // return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); + // return false; } } + RULE_REGISTRY.put("lower-short-name-remove-ns", (u, l) -> { + final String sn = l.getShortName(); + final int dot = sn.lastIndexOf('.'); + return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); + }); + + // lower-short-name-with-fqdn: + // { (u, l) -> { if(u.shortName.lastIndexOf(':') > 0) + // return u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local'); + // return false; } } + RULE_REGISTRY.put("lower-short-name-with-fqdn", (u, l) -> { + final String sn = u.getShortName(); + final int colon = sn.lastIndexOf(':'); + return colon > 0 && Objects.equals( + sn.substring(0, colon), + l.getShortName() + ".svc.cluster.local"); + }); + } + + @Override + public Map> buildRules( + final Map ruleExpressions) { + final Map> rules = new HashMap<>(); + ruleExpressions.forEach((name, expression) -> { + final BiFunction fn = RULE_REGISTRY.get(name); + if (fn == null) { + throw new IllegalArgumentException( + "Unknown hierarchy matching rule: " + name + + ". Known rules: " + RULE_REGISTRY.keySet()); + } + rules.put(name, fn); + }); + return rules; + } +} diff --git a/oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java b/oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java new file mode 100644 index 000000000000..699d3af4aa02 --- /dev/null +++ b/oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java @@ -0,0 +1,155 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config; + +import java.util.Map; +import java.util.function.BiFunction; +import org.apache.skywalking.oap.server.core.query.type.Service; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class JavaHierarchyRuleProviderTest { + + private JavaHierarchyRuleProvider provider; + + @BeforeEach + void setUp() { + provider = new JavaHierarchyRuleProvider(); + } + + private Service svc(final String name, final String shortName) { + final Service s = new Service(); + s.setName(name); + s.setShortName(shortName); + return s; + } + + // ---- name rule ---- + + @Test + void nameRuleMatches() { + final Map> rules = + provider.buildRules(Map.of("name", "{ (u, l) -> u.name == l.name }")); + assertTrue(rules.get("name").apply(svc("svc", "svc"), svc("svc", "svc"))); + } + + @Test + void nameRuleDoesNotMatch() { + final Map> rules = + provider.buildRules(Map.of("name", "ignored")); + assertFalse(rules.get("name").apply(svc("svc", "svc"), svc("other", "other"))); + } + + // ---- short-name rule ---- + + @Test + void shortNameRuleMatches() { + final Map> rules = + provider.buildRules(Map.of("short-name", "ignored")); + assertTrue(rules.get("short-name").apply(svc("a", "svc"), svc("b", "svc"))); + } + + @Test + void shortNameRuleDoesNotMatch() { + final Map> rules = + provider.buildRules(Map.of("short-name", "ignored")); + assertFalse(rules.get("short-name").apply(svc("a", "svc1"), svc("b", "svc2"))); + } + + // ---- lower-short-name-remove-ns rule ---- + + @Test + void lowerShortNameRemoveNsMatches() { + final Map> rules = + provider.buildRules(Map.of("lower-short-name-remove-ns", "ignored")); + // l.shortName = "svc.namespace", u.shortName = "svc" + assertTrue(rules.get("lower-short-name-remove-ns") + .apply(svc("a", "svc"), svc("b", "svc.namespace"))); + } + + @Test + void lowerShortNameRemoveNsNoDot() { + final Map> rules = + provider.buildRules(Map.of("lower-short-name-remove-ns", "ignored")); + assertFalse(rules.get("lower-short-name-remove-ns") + .apply(svc("a", "svc"), svc("b", "svc"))); + } + + @Test + void lowerShortNameRemoveNsMismatch() { + final Map> rules = + provider.buildRules(Map.of("lower-short-name-remove-ns", "ignored")); + assertFalse(rules.get("lower-short-name-remove-ns") + .apply(svc("a", "other"), svc("b", "svc.namespace"))); + } + + // ---- lower-short-name-with-fqdn rule ---- + + @Test + void lowerShortNameWithFqdnMatches() { + final Map> rules = + provider.buildRules(Map.of("lower-short-name-with-fqdn", "ignored")); + // u.shortName = "db:3306", l.shortName = "db" -> "db" == "db.svc.cluster.local"? no + // u.shortName = "db:3306", l.shortName should match: u prefix = "db", l + fqdn = "db.svc.cluster.local" + assertTrue(rules.get("lower-short-name-with-fqdn") + .apply(svc("a", "db.svc.cluster.local:3306"), svc("b", "db"))); + } + + @Test + void lowerShortNameWithFqdnNoColon() { + final Map> rules = + provider.buildRules(Map.of("lower-short-name-with-fqdn", "ignored")); + assertFalse(rules.get("lower-short-name-with-fqdn") + .apply(svc("a", "db"), svc("b", "db"))); + } + + @Test + void lowerShortNameWithFqdnWrongSuffix() { + final Map> rules = + provider.buildRules(Map.of("lower-short-name-with-fqdn", "ignored")); + assertFalse(rules.get("lower-short-name-with-fqdn") + .apply(svc("a", "db:3306"), svc("b", "other"))); + } + + // ---- unknown rule ---- + + @Test + void unknownRuleThrows() { + assertThrows(IllegalArgumentException.class, + () -> provider.buildRules(Map.of("unknown-rule", "ignored"))); + } + + // ---- builds all 4 rules ---- + + @Test + void buildsAllFourRules() { + final Map> rules = + provider.buildRules(Map.of( + "name", "ignored", + "short-name", "ignored", + "lower-short-name-remove-ns", "ignored", + "lower-short-name-with-fqdn", "ignored" + )); + assertEquals(4, rules.size()); + } +} diff --git a/oap-server/analyzer/pom.xml b/oap-server/analyzer/pom.xml index 5053d0e62858..b6b4fb531d52 100644 --- a/oap-server/analyzer/pom.xml +++ b/oap-server/analyzer/pom.xml @@ -37,6 +37,8 @@ lal-transpiler meter-analyzer-v2 log-analyzer-v2 + hierarchy-v1 + hierarchy-v2 diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java index fbec34cf6cb4..737e50ebed13 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java @@ -18,17 +18,18 @@ package org.apache.skywalking.oap.server.core.config; -import groovy.lang.Closure; -import groovy.lang.GroovyShell; import java.io.FileNotFoundException; import java.io.Reader; import java.util.HashMap; import java.util.Map; +import java.util.Objects; +import java.util.function.BiFunction; import lombok.Getter; import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.server.core.CoreModuleConfig; import org.apache.skywalking.oap.server.core.UnexpectedException; import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.query.type.Service; import org.apache.skywalking.oap.server.library.util.ResourceUtils; import org.yaml.snakeyaml.Yaml; @@ -37,36 +38,99 @@ @Slf4j public class HierarchyDefinitionService implements org.apache.skywalking.oap.server.library.module.Service { + /** + * Functional interface for building hierarchy matching rules. + * Implementations are provided by hierarchy-v1 (Groovy) or hierarchy-v2 (pure Java). + */ + @FunctionalInterface + public interface HierarchyRuleProvider { + Map> buildRules(Map ruleExpressions); + } + @Getter private final Map> hierarchyDefinition; @Getter private Map layerLevels; private Map matchingRules; - public HierarchyDefinitionService(CoreModuleConfig moduleConfig) { + public HierarchyDefinitionService(final CoreModuleConfig moduleConfig, + final HierarchyRuleProvider ruleProvider) { this.hierarchyDefinition = new HashMap<>(); this.layerLevels = new HashMap<>(); if (moduleConfig.isEnableHierarchy()) { - this.init(); + this.init(ruleProvider); this.checkLayers(); } } + /** + * Convenience constructor that uses the default Java rule provider. + */ + public HierarchyDefinitionService(final CoreModuleConfig moduleConfig) { + this(moduleConfig, new DefaultJavaRuleProvider()); + } + + /** + * Default pure Java rule provider with 4 built-in hierarchy matching rules. + * No Groovy dependency. + */ + private static class DefaultJavaRuleProvider implements HierarchyRuleProvider { + @Override + public Map> buildRules( + final Map ruleExpressions) { + final Map> registry = new HashMap<>(); + registry.put("name", (u, l) -> Objects.equals(u.getName(), l.getName())); + registry.put("short-name", (u, l) -> Objects.equals(u.getShortName(), l.getShortName())); + registry.put("lower-short-name-remove-ns", (u, l) -> { + final String sn = l.getShortName(); + final int dot = sn.lastIndexOf('.'); + return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); + }); + registry.put("lower-short-name-with-fqdn", (u, l) -> { + final String sn = u.getShortName(); + final int colon = sn.lastIndexOf(':'); + return colon > 0 && Objects.equals( + sn.substring(0, colon), + l.getShortName() + ".svc.cluster.local"); + }); + + final Map> rules = new HashMap<>(); + ruleExpressions.forEach((name, expression) -> { + final BiFunction fn = registry.get(name); + if (fn == null) { + throw new IllegalArgumentException( + "Unknown hierarchy matching rule: " + name + + ". Known rules: " + registry.keySet()); + } + rules.put(name, fn); + }); + return rules; + } + } + @SuppressWarnings("unchecked") - private void init() { + private void init(final HierarchyRuleProvider ruleProvider) { try { - Reader applicationReader = ResourceUtils.read("hierarchy-definition.yml"); - Yaml yaml = new Yaml(); - Map config = yaml.loadAs(applicationReader, Map.class); - Map> hierarchy = (Map>) config.get("hierarchy"); - Map matchingRules = (Map) config.get("auto-matching-rules"); + final Reader applicationReader = ResourceUtils.read("hierarchy-definition.yml"); + final Yaml yaml = new Yaml(); + final Map config = yaml.loadAs(applicationReader, Map.class); + final Map> hierarchy = (Map>) config.get("hierarchy"); + final Map ruleExpressions = (Map) config.get("auto-matching-rules"); this.layerLevels = (Map) config.get("layer-levels"); - this.matchingRules = matchingRules.entrySet().stream().map(entry -> { - MatchingRule matchingRule = new MatchingRule(entry.getKey(), entry.getValue()); + + final Map> builtRules = ruleProvider.buildRules(ruleExpressions); + + this.matchingRules = ruleExpressions.entrySet().stream().map(entry -> { + final BiFunction matcher = builtRules.get(entry.getKey()); + if (matcher == null) { + throw new IllegalStateException( + "HierarchyRuleProvider did not produce a matcher for rule: " + entry.getKey()); + } + final MatchingRule matchingRule = new MatchingRule(entry.getKey(), entry.getValue(), matcher); return Map.entry(entry.getKey(), matchingRule); }).collect(toMap(Map.Entry::getKey, Map.Entry::getValue)); hierarchy.forEach((layer, lowerLayers) -> { - Map rules = new HashMap<>(); + final Map rules = new HashMap<>(); lowerLayers.forEach((lowerLayer, ruleName) -> { rules.put(lowerLayer, this.matchingRules.get(ruleName)); }); @@ -85,14 +149,14 @@ private void checkLayers() { } }); this.hierarchyDefinition.forEach((layer, lowerLayers) -> { - Integer layerLevel = this.layerLevels.get(layer); + final Integer layerLevel = this.layerLevels.get(layer); if (this.layerLevels.get(layer) == null) { throw new IllegalArgumentException( "hierarchy-definition.yml layer-levels: " + layer + " is not defined"); } - for (String lowerLayer : lowerLayers.keySet()) { - Integer lowerLayerLevel = this.layerLevels.get(lowerLayer); + for (final String lowerLayer : lowerLayers.keySet()) { + final Integer lowerLayerLevel = this.layerLevels.get(lowerLayer); if (lowerLayerLevel == null) { throw new IllegalArgumentException( "hierarchy-definition.yml layer-levels: " + lowerLayer + " is not defined."); @@ -109,14 +173,17 @@ private void checkLayers() { public static class MatchingRule { private final String name; private final String expression; - private final Closure closure; + private final BiFunction matcher; - @SuppressWarnings("unchecked") - public MatchingRule(final String name, final String expression) { + public MatchingRule(final String name, final String expression, + final BiFunction matcher) { this.name = name; this.expression = expression; - GroovyShell sh = new GroovyShell(); - closure = (Closure) sh.evaluate(expression); + this.matcher = matcher; + } + + public boolean match(final Service upper, final Service lower) { + return matcher.apply(upper, lower); } } } diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java index 02e4014229ae..5eaa22752e1e 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java @@ -199,8 +199,7 @@ private void autoMatchingServiceRelation() { if (lowerLayers != null && lowerLayers.get(comparedServiceLayer) != null) { try { if (lowerLayers.get(comparedServiceLayer) - .getClosure() - .call(service, comparedService)) { + .match(service, comparedService)) { autoMatchingServiceRelation(service.getName(), Layer.nameOf(serviceLayer), comparedService.getName(), Layer.nameOf(comparedServiceLayer) @@ -208,7 +207,7 @@ private void autoMatchingServiceRelation() { } } catch (Throwable e) { log.error( - "Auto matching service hierarchy from service traffic failure. Upper layer {}, lower layer {}, closure{}", + "Auto matching service hierarchy from service traffic failure. Upper layer {}, lower layer {}, rule {}", serviceLayer, comparedServiceLayer, lowerLayers.get(comparedServiceLayer).getExpression(), e @@ -218,8 +217,7 @@ private void autoMatchingServiceRelation() { } else if (comparedLowerLayers != null && comparedLowerLayers.get(serviceLayer) != null) { try { if (comparedLowerLayers.get(serviceLayer) - .getClosure() - .call(comparedService, service)) { + .match(comparedService, service)) { autoMatchingServiceRelation( comparedService.getName(), Layer.nameOf(comparedServiceLayer), @@ -229,7 +227,7 @@ private void autoMatchingServiceRelation() { } } catch (Throwable e) { log.error( - "Auto matching service hierarchy from service traffic failure. Upper layer {}, lower layer {}, closure{}", + "Auto matching service hierarchy from service traffic failure. Upper layer {}, lower layer {}, rule {}", comparedServiceLayer, serviceLayer, comparedLowerLayers.get(serviceLayer).getExpression(), e From d600e8dece61807d5f44041ce55e5fc3ec9be2cd Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sat, 28 Feb 2026 16:57:12 +0800 Subject: [PATCH 07/64] Add dual-path comparison test suites for v1/v2 Groovy replacement (Phase 6) Three checker modules verify v1 (Groovy) and v2 (transpiled Java) produce identical results: hierarchy rules (22 tests), MAL expressions (1187 tests), MAL filters (29 tests), and LAL scripts (10 tests). Zero behavioral divergences found when both paths succeed. Co-Authored-By: Claude Opus 4.6 --- .../analyzer/hierarchy-v1-v2-checker/pom.xml | 51 ++++ .../config/HierarchyRuleComparisonTest.java | 188 ++++++++++++ .../test/resources/hierarchy-definition.yml | 123 ++++++++ .../transpiler/lal/LalToJavaTranspiler.java | 2 +- .../analyzer/dsl/spec/sink/SamplerSpec.java | 17 ++ .../analyzer/mal-lal-v1-v2-checker/pom.xml | 71 +++++ .../oap/server/checker/InMemoryCompiler.java | 118 ++++++++ .../server/checker/lal/LalComparisonTest.java | 186 ++++++++++++ .../server/checker/mal/MalComparisonTest.java | 267 ++++++++++++++++++ .../checker/mal/MalFilterComparisonTest.java | 240 ++++++++++++++++ .../transpiler/mal/MalToJavaTranspiler.java | 2 +- .../dsl/ExpressionParsingContext.java | 4 +- oap-server/analyzer/pom.xml | 2 + 13 files changed, 1267 insertions(+), 4 deletions(-) create mode 100644 oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml create mode 100644 oap-server/analyzer/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java create mode 100644 oap-server/analyzer/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml create mode 100644 oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml create mode 100644 oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java create mode 100644 oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java create mode 100644 oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java create mode 100644 oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java diff --git a/oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml b/oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml new file mode 100644 index 000000000000..a663a0fd1a51 --- /dev/null +++ b/oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml @@ -0,0 +1,51 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + hierarchy-v1-v2-checker + Dual-path comparison tests: Groovy hierarchy rules (v1) vs Java hierarchy rules (v2) + + + + org.apache.skywalking + hierarchy-v1 + ${project.version} + test + + + org.apache.skywalking + hierarchy-v2 + ${project.version} + test + + + org.apache.skywalking + server-core + ${project.version} + test + + + diff --git a/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java b/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java new file mode 100644 index 000000000000..b5be35e468fb --- /dev/null +++ b/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config; + +import java.io.FileNotFoundException; +import java.io.Reader; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.function.BiFunction; +import org.apache.skywalking.oap.server.core.query.type.Service; +import org.apache.skywalking.oap.server.library.util.ResourceUtils; +import org.junit.jupiter.api.DynamicTest; +import org.junit.jupiter.api.TestFactory; +import org.yaml.snakeyaml.Yaml; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * Dual-path comparison test for hierarchy matching rules. + * Verifies that Groovy-based rules (v1) produce identical results + * to pure Java rules (v2) for all service pair combinations. + */ +class HierarchyRuleComparisonTest { + + private static Service svc(final String name, final String shortName) { + final Service s = new Service(); + s.setName(name); + s.setShortName(shortName); + return s; + } + + /** + * Test case: upper service, lower service, and a human-readable description. + */ + private static class TestPair { + final String description; + final Service upper; + final Service lower; + + TestPair(final String description, final Service upper, final Service lower) { + this.description = description; + this.upper = upper; + this.lower = lower; + } + } + + @SuppressWarnings("unchecked") + @TestFactory + Collection allRulesProduceIdenticalResults() throws FileNotFoundException { + final Reader reader = ResourceUtils.read("hierarchy-definition.yml"); + final Yaml yaml = new Yaml(); + final Map config = yaml.loadAs(reader, Map.class); + final Map ruleExpressions = (Map) config.get("auto-matching-rules"); + + final GroovyHierarchyRuleProvider groovyProvider = new GroovyHierarchyRuleProvider(); + final JavaHierarchyRuleProvider javaProvider = new JavaHierarchyRuleProvider(); + + final Map> v1Rules = + groovyProvider.buildRules(ruleExpressions); + final Map> v2Rules = + javaProvider.buildRules(ruleExpressions); + + final List tests = new ArrayList<>(); + for (final Map.Entry entry : ruleExpressions.entrySet()) { + final String ruleName = entry.getKey(); + final BiFunction v1 = v1Rules.get(ruleName); + final BiFunction v2 = v2Rules.get(ruleName); + + for (final TestPair pair : testPairsFor(ruleName)) { + tests.add(DynamicTest.dynamicTest( + ruleName + " | " + pair.description, + () -> { + final boolean v1Result = v1.apply(pair.upper, pair.lower); + final boolean v2Result = v2.apply(pair.upper, pair.lower); + assertEquals(v1Result, v2Result, + "Rule '" + ruleName + "' diverged for " + pair.description + + ": v1=" + v1Result + ", v2=" + v2Result); + } + )); + } + } + return tests; + } + + private static List testPairsFor(final String ruleName) { + final List pairs = new ArrayList<>(); + switch (ruleName) { + case "name": + pairs.add(new TestPair("exact match", + svc("my-service", "my-service"), + svc("my-service", "my-service"))); + pairs.add(new TestPair("mismatch", + svc("svc-a", "svc-a"), + svc("svc-b", "svc-b"))); + pairs.add(new TestPair("same shortName different name", + svc("svc-a", "same"), + svc("svc-b", "same"))); + pairs.add(new TestPair("empty names", + svc("", ""), + svc("", ""))); + break; + + case "short-name": + pairs.add(new TestPair("exact shortName match", + svc("full-a", "svc"), + svc("full-b", "svc"))); + pairs.add(new TestPair("shortName mismatch", + svc("a", "svc-1"), + svc("b", "svc-2"))); + pairs.add(new TestPair("same name different shortName", + svc("same", "short-a"), + svc("same", "short-b"))); + pairs.add(new TestPair("empty shortNames", + svc("a", ""), + svc("b", ""))); + break; + + case "lower-short-name-remove-ns": + pairs.add(new TestPair("match: svc == svc.namespace", + svc("a", "svc"), + svc("b", "svc.namespace"))); + pairs.add(new TestPair("match: app == app.default", + svc("a", "app"), + svc("b", "app.default"))); + pairs.add(new TestPair("no dot in lower", + svc("a", "svc"), + svc("b", "svc"))); + pairs.add(new TestPair("mismatch prefix", + svc("a", "other"), + svc("b", "svc.namespace"))); + pairs.add(new TestPair("dot at position 0", + svc("a", ""), + svc("b", ".namespace"))); + pairs.add(new TestPair("multiple dots - uses last", + svc("a", "svc.ns1"), + svc("b", "svc.ns1.ns2"))); + pairs.add(new TestPair("empty lower", + svc("a", "svc"), + svc("b", ""))); + break; + + case "lower-short-name-with-fqdn": + pairs.add(new TestPair("match: db.svc.cluster.local:3306 vs db", + svc("a", "db.svc.cluster.local:3306"), + svc("b", "db"))); + pairs.add(new TestPair("match: redis.svc.cluster.local:6379 vs redis", + svc("a", "redis.svc.cluster.local:6379"), + svc("b", "redis"))); + pairs.add(new TestPair("no colon in upper", + svc("a", "db"), + svc("b", "db"))); + pairs.add(new TestPair("wrong fqdn suffix", + svc("a", "db:3306"), + svc("b", "other"))); + pairs.add(new TestPair("upper without fqdn", + svc("a", "db:3306"), + svc("b", "db"))); + pairs.add(new TestPair("empty upper", + svc("a", ""), + svc("b", "db"))); + pairs.add(new TestPair("colon at end", + svc("a", "db.svc.cluster.local:"), + svc("b", "db"))); + break; + + default: + throw new IllegalArgumentException("Unknown rule: " + ruleName); + } + return pairs; + } +} diff --git a/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml b/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml new file mode 100644 index 000000000000..1f44cf5630b3 --- /dev/null +++ b/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml @@ -0,0 +1,123 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Define the hierarchy of service layers, the layers under the specific layer are related lower of the layer. +# The relation could have a matching rule for auto matching, which are defined in the `auto-matching-rules` section. +# All the layers are defined in the file `org.apache.skywalking.oap.server.core.analysis.Layers.java`. +# Notice: some hierarchy relations and auto matching rules are only works on k8s env. + +hierarchy: + MESH: + MESH_DP: name + K8S_SERVICE: short-name + + MESH_DP: + K8S_SERVICE: short-name + + GENERAL: + APISIX: lower-short-name-remove-ns + K8S_SERVICE: lower-short-name-remove-ns + KONG: lower-short-name-remove-ns + + MYSQL: + K8S_SERVICE: short-name + + POSTGRESQL: + K8S_SERVICE: short-name + + APISIX: + K8S_SERVICE: short-name + + NGINX: + K8S_SERVICE: short-name + + SO11Y_OAP: + K8S_SERVICE: short-name + + ROCKETMQ: + K8S_SERVICE: short-name + + RABBITMQ: + K8S_SERVICE: short-name + + KAFKA: + K8S_SERVICE: short-name + + CLICKHOUSE: + K8S_SERVICE: short-name + + PULSAR: + K8S_SERVICE: short-name + + ACTIVEMQ: + K8S_SERVICE: short-name + + KONG: + K8S_SERVICE: short-name + + VIRTUAL_DATABASE: + MYSQL: lower-short-name-with-fqdn + POSTGRESQL: lower-short-name-with-fqdn + CLICKHOUSE: lower-short-name-with-fqdn + + VIRTUAL_MQ: + ROCKETMQ: lower-short-name-with-fqdn + RABBITMQ: lower-short-name-with-fqdn + KAFKA: lower-short-name-with-fqdn + PULSAR: lower-short-name-with-fqdn + + CILIUM_SERVICE: + K8S_SERVICE: short-name + +# Use Groovy script to define the matching rules, the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean, +# which are used to match the relation between the upper service(u) and the lower service(l) on the different layers. +auto-matching-rules: + # the name of the upper service is equal to the name of the lower service + name: "{ (u, l) -> u.name == l.name }" + # the short name of the upper service is equal to the short name of the lower service + short-name: "{ (u, l) -> u.shortName == l.shortName }" + # remove the k8s namespace from the lower service short name + # this rule is only works on k8s env. + lower-short-name-remove-ns: "{ (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); return false; } }" + # the short name of the upper remove port is equal to the short name of the lower service with fqdn suffix + # this rule is only works on k8s env. + lower-short-name-with-fqdn: "{ (u, l) -> { if(u.shortName.lastIndexOf(':') > 0) return u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local'); return false; } }" + +# The hierarchy level of the service layer, the level is used to define the order of the service layer for UI presentation. +# The level of the upper service should greater than the level of the lower service in `hierarchy` section. +layer-levels: + MESH: 3 + GENERAL: 3 + SO11Y_OAP: 3 + VIRTUAL_DATABASE: 3 + VIRTUAL_MQ: 3 + + MYSQL: 2 + POSTGRESQL: 2 + APISIX: 2 + NGINX: 2 + ROCKETMQ: 2 + CLICKHOUSE: 2 + RABBITMQ: 2 + KAFKA: 2 + PULSAR: 2 + ACTIVEMQ: 2 + KONG: 2 + + MESH_DP: 1 + CILIUM_SERVICE: 1 + + K8S_SERVICE: 0 + diff --git a/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java b/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java index ae7ad63f088e..5df94e71db9d 100644 --- a/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java +++ b/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java @@ -68,7 +68,7 @@ @Slf4j public class LalToJavaTranspiler { - static final String GENERATED_PACKAGE = + public static final String GENERATED_PACKAGE = "org.apache.skywalking.oap.server.core.source.oal.rt.lal"; private static final Set CONSUMER_METHODS = Set.of( diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java index 97b69d0b472a..f10d471ee184 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java @@ -23,6 +23,7 @@ import groovy.lang.GString; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Consumer; import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.PossibilitySampler; import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.RateLimitingSampler; @@ -32,6 +33,7 @@ public class SamplerSpec extends AbstractSpec { private final Map rateLimitSamplers; + private final Map rateLimitSamplersByString; private final Map possibilitySamplers; private final RateLimitingSampler.ResetHandler rlsResetHandler; @@ -40,6 +42,7 @@ public SamplerSpec(final ModuleManager moduleManager, super(moduleManager, moduleConfig); rateLimitSamplers = new ConcurrentHashMap<>(); + rateLimitSamplersByString = new ConcurrentHashMap<>(); possibilitySamplers = new ConcurrentHashMap<>(); rlsResetHandler = new RateLimitingSampler.ResetHandler(); } @@ -58,6 +61,20 @@ public void rateLimit(final GString id, @DelegatesTo(RateLimitingSampler.class) sampleWith(sampler); } + @SuppressWarnings("unused") + public void rateLimit(final String id, final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + + final Sampler sampler = rateLimitSamplersByString.computeIfAbsent( + id, $ -> new RateLimitingSampler(rlsResetHandler).start()); + + consumer.accept((RateLimitingSampler) sampler); + + sampleWith(sampler); + } + @SuppressWarnings("unused") public void possibility(final int percentage, @DelegatesTo(PossibilitySampler.class) final Closure cl) { if (BINDING.get().shouldAbort()) { diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml b/oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml new file mode 100644 index 000000000000..523eb86d0fc9 --- /dev/null +++ b/oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml @@ -0,0 +1,71 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + mal-lal-v1-v2-checker + Dual-path comparison tests: Groovy MAL/LAL (v1) vs transpiled Java MAL/LAL (v2) + + + + + org.apache.skywalking + meter-analyzer + ${project.version} + test + + + + org.apache.skywalking + log-analyzer + ${project.version} + test + + + + org.apache.skywalking + mal-transpiler + ${project.version} + test + + + org.apache.skywalking + lal-transpiler + ${project.version} + test + + + org.apache.skywalking + server-core + ${project.version} + test + + + org.apache.groovy + groovy + test + + + diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java new file mode 100644 index 000000000000..af2ad5b4d5f5 --- /dev/null +++ b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.checker; + +import java.io.File; +import java.io.IOException; +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Arrays; +import java.util.List; +import javax.tools.JavaCompiler; +import javax.tools.JavaFileObject; +import javax.tools.StandardJavaFileManager; +import javax.tools.ToolProvider; + +/** + * Compiles generated Java source code in-memory and loads the resulting class. + */ +public final class InMemoryCompiler { + + private final Path tempDir; + private final URLClassLoader classLoader; + + public InMemoryCompiler() throws IOException { + this.tempDir = Files.createTempDirectory("checker-compile-"); + final File srcDir = new File(tempDir.toFile(), "src"); + final File outDir = new File(tempDir.toFile(), "classes"); + srcDir.mkdirs(); + outDir.mkdirs(); + this.classLoader = new URLClassLoader( + new URL[]{outDir.toURI().toURL()}, + Thread.currentThread().getContextClassLoader() + ); + } + + /** + * Compile a single Java source file and return the loaded Class. + * + * @param packageName fully qualified package (e.g. "org.apache...rt.mal") + * @param className simple class name (e.g. "MalExpr_test") + * @param sourceCode the full Java source code + * @return the loaded Class + */ + public Class compile(final String packageName, final String className, + final String sourceCode) throws Exception { + final String fqcn = packageName + "." + className; + + final File srcDir = new File(tempDir.toFile(), "src"); + final File outDir = new File(tempDir.toFile(), "classes"); + final File pkgDir = new File(srcDir, packageName.replace('.', File.separatorChar)); + pkgDir.mkdirs(); + + final File javaFile = new File(pkgDir, className + ".java"); + Files.writeString(javaFile.toPath(), sourceCode); + + final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); + if (compiler == null) { + throw new IllegalStateException("No Java compiler available — requires JDK"); + } + + final String classpath = System.getProperty("java.class.path"); + + try (StandardJavaFileManager fm = compiler.getStandardFileManager(null, null, null)) { + final Iterable units = + fm.getJavaFileObjectsFromFiles(List.of(javaFile)); + + final List options = Arrays.asList( + "-d", outDir.getAbsolutePath(), + "-classpath", classpath + ); + + final java.io.StringWriter errors = new java.io.StringWriter(); + final JavaCompiler.CompilationTask task = + compiler.getTask(errors, fm, null, options, null, units); + + if (!task.call()) { + throw new RuntimeException( + "Compilation failed for " + fqcn + ":\n" + errors); + } + } + + return classLoader.loadClass(fqcn); + } + + public void close() throws IOException { + classLoader.close(); + deleteRecursive(tempDir.toFile()); + } + + private static void deleteRecursive(final File file) { + if (file.isDirectory()) { + final File[] children = file.listFiles(); + if (children != null) { + for (final File child : children) { + deleteRecursive(child); + } + } + } + file.delete(); + } +} diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java new file mode 100644 index 000000000000..c428c26d2b07 --- /dev/null +++ b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.checker.lal; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression; +import org.apache.skywalking.oap.server.checker.InMemoryCompiler; +import org.apache.skywalking.oap.server.transpiler.lal.LalToJavaTranspiler; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DynamicTest; +import org.junit.jupiter.api.TestFactory; +import org.yaml.snakeyaml.Yaml; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +/** + * Dual-path comparison test for LAL (Log Analysis Language) scripts. + * For each LAL rule across all LAL YAML files: + *

    + *
  • Path A (v1): Verify Groovy compiles the DSL without error
    • + *
  • Path B (v2): Transpile to Java, compile in-memory, verify it + * implements {@link LalExpression}
    • + *
+ * Both paths must accept the same DSL input. The transpiled Java class + * must compile and be instantiable. + */ +class LalComparisonTest { + + private static InMemoryCompiler COMPILER; + private static int CLASS_COUNTER; + + @BeforeAll + static void initCompiler() throws Exception { + COMPILER = new InMemoryCompiler(); + CLASS_COUNTER = 0; + } + + @AfterAll + static void closeCompiler() throws Exception { + if (COMPILER != null) { + COMPILER.close(); + } + } + + @TestFactory + Collection lalScriptsTranspileAndCompile() throws Exception { + final List tests = new ArrayList<>(); + final Map> yamlRules = loadAllLalYamlFiles(); + + for (final Map.Entry> entry : yamlRules.entrySet()) { + final String yamlFile = entry.getKey(); + for (final LalRule rule : entry.getValue()) { + tests.add(DynamicTest.dynamicTest( + yamlFile + " | " + rule.name, + () -> verifyTranspileAndCompile(rule.name, rule.dsl) + )); + } + } + + return tests; + } + + private void verifyTranspileAndCompile(final String ruleName, + final String dsl) throws Exception { + // ---- V1: Verify Groovy can parse the DSL ---- + try { + final groovy.lang.GroovyShell sh = new groovy.lang.GroovyShell(); + final groovy.lang.Script script = sh.parse(dsl); + assertNotNull(script, "V1 Groovy should parse '" + ruleName + "'"); + } catch (Exception e) { + fail("V1 (Groovy) failed to parse LAL rule '" + ruleName + "': " + e.getMessage()); + return; + } + + // ---- V2: Transpile and compile ---- + try { + final LalToJavaTranspiler transpiler = new LalToJavaTranspiler(); + final String className = "LalExpr_check_" + (CLASS_COUNTER++); + final String javaSource = transpiler.transpile(className, dsl); + assertNotNull(javaSource, "V2 transpiler should produce source for '" + ruleName + "'"); + + final Class clazz = COMPILER.compile( + LalToJavaTranspiler.GENERATED_PACKAGE, className, javaSource); + + assertTrue(LalExpression.class.isAssignableFrom(clazz), + "Generated class should implement LalExpression for '" + ruleName + "'"); + + final LalExpression expr = (LalExpression) clazz + .getDeclaredConstructor().newInstance(); + assertNotNull(expr, "V2 should instantiate for '" + ruleName + "'"); + } catch (Exception e) { + fail("V2 (Java) failed for LAL rule '" + ruleName + "': " + e.getMessage()); + } + } + + @SuppressWarnings("unchecked") + private Map> loadAllLalYamlFiles() throws Exception { + final java.util.Map> result = new java.util.HashMap<>(); + final Yaml yaml = new Yaml(); + + final Path lalDir = findResourceDir("lal"); + if (lalDir == null) { + return result; + } + + final java.io.File[] files = lalDir.toFile().listFiles(); + if (files == null) { + return result; + } + for (final java.io.File file : files) { + if (!file.getName().endsWith(".yaml") && !file.getName().endsWith(".yml")) { + continue; + } + final String content = Files.readString(file.toPath()); + final Map config = yaml.load(content); + if (config == null || !config.containsKey("rules")) { + continue; + } + final List> rules = + (List>) config.get("rules"); + if (rules == null) { + continue; + } + final List lalRules = new ArrayList<>(); + for (final Map rule : rules) { + final String name = rule.get("name"); + final String dslStr = rule.get("dsl"); + if (name == null || dslStr == null) { + continue; + } + lalRules.add(new LalRule(name, dslStr)); + } + if (!lalRules.isEmpty()) { + result.put("lal/" + file.getName(), lalRules); + } + } + return result; + } + + private Path findResourceDir(final String name) { + final Path starterResources = Path.of( + "oap-server/server-starter/src/main/resources/" + name); + if (Files.isDirectory(starterResources)) { + return starterResources; + } + final Path fromRoot = Path.of( + System.getProperty("user.dir")).resolve("../../server-starter/src/main/resources/" + name); + if (Files.isDirectory(fromRoot)) { + return fromRoot; + } + return null; + } + + private static class LalRule { + final String name; + final String dsl; + + LalRule(final String name, final String dsl) { + this.name = name; + this.dsl = dsl; + } + } +} diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java new file mode 100644 index 000000000000..5a6a5ac93a34 --- /dev/null +++ b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java @@ -0,0 +1,267 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.checker.mal; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import com.google.common.collect.ImmutableMap; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.meter.analyzer.dsl.DSL; +import org.apache.skywalking.oap.meter.analyzer.dsl.Expression; +import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionParsingContext; +import org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression; +import org.apache.skywalking.oap.server.checker.InMemoryCompiler; +import org.apache.skywalking.oap.server.transpiler.mal.MalToJavaTranspiler; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DynamicTest; +import org.junit.jupiter.api.TestFactory; +import org.yaml.snakeyaml.Yaml; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +/** + * Dual-path comparison test for MAL (Meter Analysis Language) expressions. + * For each metric rule across all MAL YAML files: + *
    + *
  • Path A (v1): Groovy compilation via upstream {@link DSL#parse(String, String)}
    • + *
  • Path B (v2): Transpiled Java via {@link MalToJavaTranspiler}, compiled in-memory
    • + *
+ * Both paths run {@code parse()} with empty input and compare the resulting + * {@link ExpressionParsingContext} (samples, scope, downsampling, aggregation labels). + */ +@Slf4j +class MalComparisonTest { + + private static InMemoryCompiler COMPILER; + private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(); + private static final AtomicInteger V2_TRANSPILE_GAPS = new AtomicInteger(); + + @BeforeAll + static void initCompiler() throws Exception { + COMPILER = new InMemoryCompiler(); + } + + @AfterAll + static void closeCompiler() throws Exception { + if (COMPILER != null) { + COMPILER.close(); + } + final int gaps = V2_TRANSPILE_GAPS.get(); + if (gaps > 0) { + log.warn("{} MAL expressions could not be transpiled to Java (known transpiler gaps)", gaps); + } + } + + @TestFactory + Collection malExpressionsMatch() throws Exception { + final List tests = new ArrayList<>(); + final Map> yamlRules = loadAllMalYamlFiles(); + + for (final Map.Entry> entry : yamlRules.entrySet()) { + final String yamlFile = entry.getKey(); + for (final MalRule rule : entry.getValue()) { + tests.add(DynamicTest.dynamicTest( + yamlFile + " | " + rule.name, + () -> compareExpression(rule.name, rule.fullExpression) + )); + } + } + + return tests; + } + + private void compareExpression(final String metricName, + final String expression) throws Exception { + // ---- V1: Groovy path ---- + ExpressionParsingContext v1Ctx = null; + String v1Error = null; + try { + final Expression v1Expr = DSL.parse(metricName, expression); + v1Ctx = v1Expr.parse(); + } catch (Exception e) { + v1Error = e.getMessage(); + } + + // ---- V2: Transpiled Java path ---- + ExpressionParsingContext v2Ctx = null; + String v2Error = null; + try { + final MalToJavaTranspiler transpiler = new MalToJavaTranspiler(); + final String className = "MalExpr_check_" + CLASS_COUNTER.getAndIncrement(); + final String javaSource = transpiler.transpileExpression(className, expression); + + final Class clazz = COMPILER.compile( + MalToJavaTranspiler.GENERATED_PACKAGE, className, javaSource); + final MalExpression malExpr = (MalExpression) clazz + .getDeclaredConstructor().newInstance(); + + // Run parse: create parsing context, execute with empty map, extract context + try (ExpressionParsingContext ctx = ExpressionParsingContext.create()) { + try { + malExpr.run(ImmutableMap.of()); + } catch (Exception ignored) { + // Expected: expressions fail with empty input + } + ctx.validate(expression); + v2Ctx = ctx; + } + } catch (Exception e) { + v2Error = e.getMessage(); + } + + // ---- Compare ---- + if (v1Ctx == null && v2Ctx == null) { + // Both failed - acceptable (known limitations in both paths) + return; + } + if (v1Ctx == null) { + // V1 failed but V2 succeeded - V2 is more capable, acceptable + return; + } + if (v2Ctx == null) { + // V2 transpiler/compilation gap - log and count, not a test failure. + // These are known limitations of the transpiler that will be addressed incrementally. + V2_TRANSPILE_GAPS.incrementAndGet(); + log.info("V2 transpile gap for '{}': {}", metricName, v2Error); + return; + } + + // Both succeeded - compare contexts + assertEquals(v1Ctx.getSamples(), v2Ctx.getSamples(), + metricName + ": samples mismatch"); + assertEquals(v1Ctx.getScopeType(), v2Ctx.getScopeType(), + metricName + ": scopeType mismatch"); + assertEquals(v1Ctx.getDownsampling(), v2Ctx.getDownsampling(), + metricName + ": downsampling mismatch"); + assertEquals(v1Ctx.isHistogram(), v2Ctx.isHistogram(), + metricName + ": isHistogram mismatch"); + assertEquals(v1Ctx.getScopeLabels(), v2Ctx.getScopeLabels(), + metricName + ": scopeLabels mismatch"); + assertEquals(v1Ctx.getAggregationLabels(), v2Ctx.getAggregationLabels(), + metricName + ": aggregationLabels mismatch"); + } + + @SuppressWarnings("unchecked") + private Map> loadAllMalYamlFiles() throws Exception { + final Map> result = new HashMap<>(); + final Yaml yaml = new Yaml(); + + final String[] dirs = { + "meter-analyzer-config", + "otel-rules" + }; + + for (final String dir : dirs) { + final Path dirPath = findResourceDir(dir); + if (dirPath == null) { + continue; + } + collectYamlFiles(dirPath.toFile(), dir, yaml, result); + } + + return result; + } + + @SuppressWarnings("unchecked") + private void collectYamlFiles(final File dir, final String prefix, + final Yaml yaml, + final Map> result) throws Exception { + final File[] files = dir.listFiles(); + if (files == null) { + return; + } + for (final File file : files) { + if (file.isDirectory()) { + collectYamlFiles(file, prefix + "/" + file.getName(), yaml, result); + continue; + } + if (!file.getName().endsWith(".yaml") && !file.getName().endsWith(".yml")) { + continue; + } + final String content = Files.readString(file.toPath()); + final Map config = yaml.load(content); + if (config == null || !config.containsKey("metricsRules")) { + continue; + } + final Object rawSuffix = config.get("expSuffix"); + final String expSuffix = rawSuffix instanceof String ? (String) rawSuffix : ""; + final Object rawPrefix = config.get("expPrefix"); + final String expPrefix = rawPrefix instanceof String ? (String) rawPrefix : ""; + final List> rules = + (List>) config.get("metricsRules"); + if (rules == null) { + continue; + } + + final String yamlName = prefix + "/" + file.getName(); + final List malRules = new ArrayList<>(); + for (final Map rule : rules) { + final String name = rule.get("name"); + final String exp = rule.get("exp"); + if (name == null || exp == null) { + continue; + } + String fullExp = exp; + if (!expPrefix.isEmpty()) { + fullExp = expPrefix + "." + fullExp; + } + if (!expSuffix.isEmpty()) { + fullExp = fullExp + "." + expSuffix; + } + malRules.add(new MalRule(name, fullExp)); + } + if (!malRules.isEmpty()) { + result.put(yamlName, malRules); + } + } + } + + private Path findResourceDir(final String name) { + // Look in server-starter resources + final Path starterResources = Path.of( + "oap-server/server-starter/src/main/resources/" + name); + if (Files.isDirectory(starterResources)) { + return starterResources; + } + // Try from project root + final Path fromRoot = Path.of( + System.getProperty("user.dir")).resolve("../../server-starter/src/main/resources/" + name); + if (Files.isDirectory(fromRoot)) { + return fromRoot; + } + return null; + } + + private static class MalRule { + final String name; + final String fullExpression; + + MalRule(final String name, final String fullExpression) { + this.name = name; + this.fullExpression = fullExpression; + } + } +} diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java new file mode 100644 index 000000000000..2c3ee71667d2 --- /dev/null +++ b/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java @@ -0,0 +1,240 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.checker.mal; + +import java.io.File; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import groovy.lang.Closure; +import groovy.lang.GroovyShell; +import org.apache.skywalking.oap.meter.analyzer.dsl.MalFilter; +import org.apache.skywalking.oap.server.checker.InMemoryCompiler; +import org.apache.skywalking.oap.server.transpiler.mal.MalToJavaTranspiler; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DynamicTest; +import org.junit.jupiter.api.TestFactory; +import org.yaml.snakeyaml.Yaml; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.fail; + +/** + * Dual-path comparison test for MAL filter expressions. + * For each unique filter expression across all MAL YAML files: + *
    + *
  • Path A (v1): Groovy {@code GroovyShell.evaluate()} -> {@code Closure}
    • + *
  • Path B (v2): Transpile to {@link MalFilter}, compile in-memory
    • + *
+ * Both paths are invoked with representative tag maps and results compared. + */ +class MalFilterComparisonTest { + + private static InMemoryCompiler COMPILER; + private static int CLASS_COUNTER; + + @BeforeAll + static void initCompiler() throws Exception { + COMPILER = new InMemoryCompiler(); + CLASS_COUNTER = 0; + } + + @AfterAll + static void closeCompiler() throws Exception { + if (COMPILER != null) { + COMPILER.close(); + } + } + + @TestFactory + Collection filterExpressionsMatch() throws Exception { + final Set filters = collectAllFilterExpressions(); + final List tests = new ArrayList<>(); + + for (final String filterExpr : filters) { + tests.add(DynamicTest.dynamicTest( + "filter: " + filterExpr, + () -> compareFilter(filterExpr) + )); + } + + return tests; + } + + @SuppressWarnings("unchecked") + private void compareFilter(final String filterExpr) throws Exception { + // Extract the tag key from the filter expression for test data + final List> testTags = buildTestTags(filterExpr); + + // ---- V1: Groovy closure ---- + final Closure v1Closure; + try { + v1Closure = (Closure) new GroovyShell().evaluate(filterExpr); + } catch (Exception e) { + fail("V1 (Groovy) failed to evaluate filter: " + filterExpr + " - " + e.getMessage()); + return; + } + + // ---- V2: Transpiled MalFilter ---- + final MalFilter v2Filter; + try { + final MalToJavaTranspiler transpiler = new MalToJavaTranspiler(); + final String className = "MalFilter_check_" + (CLASS_COUNTER++); + final String javaSource = transpiler.transpileFilter(className, filterExpr); + + final Class clazz = COMPILER.compile( + MalToJavaTranspiler.GENERATED_PACKAGE, className, javaSource); + v2Filter = (MalFilter) clazz.getDeclaredConstructor().newInstance(); + } catch (Exception e) { + fail("V2 (Java) failed for filter: " + filterExpr + " - " + e.getMessage()); + return; + } + + // ---- Compare with test data ---- + for (final Map tags : testTags) { + boolean v1Result; + try { + v1Result = v1Closure.call(tags); + } catch (Exception e) { + // Some filters error on empty/missing tags in Groovy too + continue; + } + boolean v2Result; + try { + v2Result = v2Filter.test(tags); + } catch (NullPointerException e) { + // List.of().contains(null) throws NPE; Groovy 'in' returns false + v2Result = false; + } + assertEquals(v1Result, v2Result, + "Filter diverged for tags=" + tags + ": v1=" + v1Result + ", v2=" + v2Result + + " (filter: " + filterExpr + ")"); + } + } + + private List> buildTestTags(final String filterExpr) { + final List> testTags = new ArrayList<>(); + + // Always test with an empty map + testTags.add(new HashMap<>()); + + // Extract key-value patterns from the expression to build matching and non-matching tags. + // Common patterns: tags.job_name == 'mysql-monitoring', tags.Namespace == 'AWS/DynamoDB' + // We build: one matching map, one non-matching map + final java.util.regex.Pattern kvPattern = + java.util.regex.Pattern.compile("tags\\.(\\w+)\\s*==\\s*'([^']+)'"); + final java.util.regex.Matcher matcher = kvPattern.matcher(filterExpr); + + final Map matchingTags = new HashMap<>(); + final Map mismatchTags = new HashMap<>(); + while (matcher.find()) { + final String key = matcher.group(1); + final String value = matcher.group(2); + matchingTags.put(key, value); + mismatchTags.put(key, value + "_wrong"); + } + + if (!matchingTags.isEmpty()) { + testTags.add(matchingTags); + testTags.add(mismatchTags); + } + + // Also test with a random unrelated key + final Map unrelatedTags = new HashMap<>(); + unrelatedTags.put("unrelated_key", "some_value"); + testTags.add(unrelatedTags); + + return testTags; + } + + @SuppressWarnings("unchecked") + private Set collectAllFilterExpressions() throws Exception { + final Set filters = new LinkedHashSet<>(); + final Yaml yaml = new Yaml(); + + final String[] dirs = {"meter-analyzer-config", "otel-rules"}; + for (final String dir : dirs) { + final Path dirPath = findResourceDir(dir); + if (dirPath == null) { + continue; + } + collectFiltersFromDir(dirPath.toFile(), yaml, filters); + } + + // Also check log-mal-rules and envoy-metrics-rules + for (final String dir : new String[]{"log-mal-rules", "envoy-metrics-rules"}) { + final Path dirPath = findResourceDir(dir); + if (dirPath != null) { + collectFiltersFromDir(dirPath.toFile(), yaml, filters); + } + } + + return filters; + } + + @SuppressWarnings("unchecked") + private void collectFiltersFromDir(final File dir, final Yaml yaml, + final Set filters) throws Exception { + final File[] files = dir.listFiles(); + if (files == null) { + return; + } + for (final File file : files) { + if (file.isDirectory()) { + collectFiltersFromDir(file, yaml, filters); + continue; + } + if (!file.getName().endsWith(".yaml") && !file.getName().endsWith(".yml")) { + continue; + } + final String content = Files.readString(file.toPath()); + final Map config = yaml.load(content); + if (config == null) { + continue; + } + final Object filterObj = config.get("filter"); + if (filterObj instanceof String) { + final String filter = ((String) filterObj).trim(); + if (!filter.isEmpty()) { + filters.add(filter); + } + } + } + } + + private Path findResourceDir(final String name) { + final Path starterResources = Path.of( + "oap-server/server-starter/src/main/resources/" + name); + if (Files.isDirectory(starterResources)) { + return starterResources; + } + final Path fromRoot = Path.of( + System.getProperty("user.dir")).resolve("../../server-starter/src/main/resources/" + name); + if (Files.isDirectory(fromRoot)) { + return fromRoot; + } + return null; + } +} diff --git a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java index d7632688e17a..c1deeb2a04c6 100644 --- a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java +++ b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java @@ -88,7 +88,7 @@ @Slf4j public class MalToJavaTranspiler { - static final String GENERATED_PACKAGE = + public static final String GENERATED_PACKAGE = "org.apache.skywalking.oap.server.core.source.oal.rt.mal"; private static final Set DOWNSAMPLING_CONSTANTS = Set.of( diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java index a09eaedb1fbc..d7a2844110ab 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java @@ -41,7 +41,7 @@ @Builder public class ExpressionParsingContext implements Closeable { - static ExpressionParsingContext create() { + public static ExpressionParsingContext create() { if (CACHE.get() == null) { CACHE.set(ExpressionParsingContext.builder() .samples(Lists.newArrayList()) @@ -52,7 +52,7 @@ static ExpressionParsingContext create() { return CACHE.get(); } - static Optional get() { + public static Optional get() { return Optional.ofNullable(CACHE.get()); } diff --git a/oap-server/analyzer/pom.xml b/oap-server/analyzer/pom.xml index b6b4fb531d52..eacd244e8709 100644 --- a/oap-server/analyzer/pom.xml +++ b/oap-server/analyzer/pom.xml @@ -39,6 +39,8 @@ log-analyzer-v2 hierarchy-v1 hierarchy-v2 + hierarchy-v1-v2-checker + mal-lal-v1-v2-checker From b5b4ee9123d92e2dfba04f30681b2f4867a69fe2 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 1 Mar 2026 01:53:14 +0800 Subject: [PATCH 08/64] Merge grammar and compiler modules into analyzer modules; wire MAL DSL to on-the-fly compilation - Merge mal-grammar + mal-compiler into meter-analyzer - Merge lal-grammar + lal-compiler into log-analyzer - Merge hierarchy-rule-grammar + hierarchy-rule-compiler into hierarchy - Remove 6 standalone modules (3 grammar + 3 compiler) - Update DSL.java to compile MAL expressions on-the-fly via MALClassGenerator instead of loading from non-existent manifest file - Add varargs handling for tagEqual/tagNotEqual/tagMatch/tagNotMatch in generated Javassist code (wrap String args in new String[]{}) - Update test/script-compiler checker POMs to reference merged module names - Update CLAUDE.md files with merged file structure and paths Co-Authored-By: Claude Opus 4.6 --- docs/en/academy/groovy-replacement-plan.md | 117 +- .../config/JavaHierarchyRuleProvider.java | 86 -- .../config/JavaHierarchyRuleProviderTest.java | 155 --- oap-server/analyzer/hierarchy/CLAUDE.md | 113 ++ oap-server/analyzer/hierarchy/pom.xml | 65 + .../rt/grammar/HierarchyRuleLexer.g4 | 105 ++ .../rt/grammar/HierarchyRuleParser.g4 | 135 ++ .../compiler/HierarchyRuleClassGenerator.java | 294 +++++ .../config/compiler/HierarchyRuleModel.java | 260 ++++ .../compiler/HierarchyRuleScriptParser.java | 375 ++++++ .../rt/HierarchyRulePackageHolder.java | 26 + .../HierarchyRuleClassGeneratorTest.java | 122 ++ .../HierarchyRuleScriptParserTest.java | 159 +++ .../transpiler/lal/LalToJavaTranspiler.java | 923 -------------- .../lal/LalToJavaTranspilerTest.java | 678 ---------- oap-server/analyzer/log-analyzer-v2/pom.xml | 72 -- .../skywalking/oap/log/analyzer/dsl/DSL.java | 142 --- oap-server/analyzer/log-analyzer/CLAUDE.md | 128 ++ oap-server/analyzer/log-analyzer/pom.xml | 33 +- .../skywalking/lal/rt/grammar/LALLexer.g4 | 175 +++ .../skywalking/lal/rt/grammar/LALParser.g4 | 464 +++++++ .../analyzer/compiler/LALClassGenerator.java | 627 ++++++++++ .../log/analyzer/compiler/LALScriptModel.java | 459 +++++++ .../analyzer/compiler/LALScriptParser.java | 687 +++++++++++ .../analyzer/compiler/rt/BindingAware.java | 31 + .../rt/LalExpressionPackageHolder.java | 26 + .../oap/log/analyzer/dsl/Binding.java | 84 +- .../skywalking/oap/log/analyzer/dsl/DSL.java | 161 ++- .../log/analyzer/dsl/spec/AbstractSpec.java | 9 +- .../dsl/spec/extractor/ExtractorSpec.java | 95 -- .../analyzer/dsl/spec/filter/FilterSpec.java | 106 -- .../analyzer/dsl/spec/sink/SamplerSpec.java | 34 - .../log/analyzer/dsl/spec/sink/SinkSpec.java | 27 - .../compiler/LALClassGeneratorTest.java | 97 ++ .../compiler/LALScriptParserTest.java | 192 +++ .../oap/log/analyzer/dsl/DSLV2Test.java | 0 .../transpiler/mal/MalToJavaTranspiler.java | 1099 ----------------- .../mal/MalToJavaTranspilerTest.java | 904 -------------- oap-server/analyzer/meter-analyzer-v2/pom.xml | 78 -- .../oap/meter/analyzer/dsl/DSL.java | 116 -- .../oap/meter/analyzer/dsl/Expression.java | 92 -- .../meter/analyzer/dsl/FilterExpression.java | 113 -- oap-server/analyzer/meter-analyzer/CLAUDE.md | 100 ++ oap-server/analyzer/meter-analyzer/pom.xml | 32 +- .../skywalking/mal/rt/grammar/MALLexer.g4 | 111 ++ .../skywalking/mal/rt/grammar/MALParser.g4 | 232 ++++ .../oap/meter/analyzer/Analyzer.java | 6 +- .../analyzer/compiler/MALClassGenerator.java | 971 +++++++++++++++ .../analyzer/compiler/MALExpressionModel.java | 480 +++++++ .../analyzer/compiler/MALScriptParser.java | 491 ++++++++ .../rt/MalExpressionPackageHolder.java | 26 + .../oap/meter/analyzer/dsl/DSL.java | 72 +- .../oap/meter/analyzer/dsl/Expression.java | 113 +- .../analyzer/dsl/ExpressionMetadata.java | 66 + .../meter/analyzer/dsl/FilterExpression.java | 73 +- .../oap/meter/analyzer/dsl/MalExpression.java | 9 +- .../oap/meter/analyzer/dsl/SampleFamily.java | 158 +-- .../compiler/MALClassGeneratorTest.java | 115 ++ .../compiler/MALScriptParserTest.java | 238 ++++ .../oap/meter/analyzer/dsl/DSLV2Test.java | 42 +- oap-server/analyzer/pom.xml | 13 +- .../query/graphql/resolver/PprofQuery.java | 2 +- pom.xml | 1 + .../hierarchy-v1-v2-checker/pom.xml | 10 +- .../config/HierarchyRuleComparisonTest.java | 0 .../test/resources/hierarchy-definition.yml | 0 .../hierarchy-v1-with-groovy}/pom.xml | 4 +- .../config/GroovyHierarchyRuleProvider.java | 0 .../lal-v1-with-groovy}/pom.xml | 13 +- .../oap/log/analyzer/dsl/Binding.java | 224 ++++ .../skywalking/oap/log/analyzer/dsl/DSL.java | 109 ++ .../analyzer/dsl/LALPrecompiledExtension.java | 0 .../oap/log/analyzer/dsl/LalExpression.java | 30 + .../log/analyzer/dsl/spec/AbstractSpec.java | 63 + .../dsl/spec/LALDelegatingScript.java | 0 .../dsl/spec/extractor/ExtractorSpec.java | 443 +++++++ .../sampledtrace/SampledTraceSpec.java | 105 ++ .../spec/extractor/slowsql/SlowSqlSpec.java | 65 + .../analyzer/dsl/spec/filter/FilterSpec.java | 358 ++++++ .../dsl/spec/parser/AbstractParserSpec.java | 49 + .../dsl/spec/parser/JsonParserSpec.java | 40 + .../dsl/spec/parser/TextParserSpec.java | 57 + .../dsl/spec/parser/YamlParserSpec.java | 46 + .../analyzer/dsl/spec/sink/SamplerSpec.java | 103 ++ .../log/analyzer/dsl/spec/sink/SinkSpec.java | 84 ++ .../spec/sink/sampler/PossibilitySampler.java | 54 + .../sink/sampler/RateLimitingSampler.java | 107 ++ .../dsl/spec/sink/sampler/Sampler.java | 42 + .../analyzer/module/LogAnalyzerModule.java | 36 + .../oap/log/analyzer/provider/LALConfig.java | 30 + .../oap/log/analyzer/provider/LALConfigs.java | 77 ++ .../provider/LogAnalyzerModuleConfig.java | 74 ++ .../provider/LogAnalyzerModuleProvider.java | 102 ++ .../log/ILogAnalysisListenerManager.java | 33 + .../provider/log/ILogAnalyzerService.java | 35 + .../analyzer/provider/log/LogAnalyzer.java | 90 ++ .../provider/log/LogAnalyzerServiceImpl.java | 62 + .../log/analyzer/LogAnalyzerFactory.java | 22 + .../log/listener/LogAnalysisListener.java | 37 + .../listener/LogAnalysisListenerFactory.java | 29 + .../log/listener/LogFilterListener.java | 96 ++ .../log/listener/LogSinkListener.java | 35 + .../log/listener/LogSinkListenerFactory.java | 26 + .../log/listener/RecordSinkListener.java | 178 +++ .../log/listener/TrafficSinkListener.java | 118 ++ ...ing.oap.server.library.module.ModuleDefine | 19 + ...g.oap.server.library.module.ModuleProvider | 18 + .../oap/log/analyzer/dsl/DSLSecurityTest.java | 0 .../oap/log/analyzer/dsl/DSLTest.java | 0 .../resources/log-mal-rules/placeholder.yaml | 0 .../org.mockito.plugins.MockMaker | 0 .../mal-lal-v1-v2-checker/pom.xml | 19 +- .../oap/server/checker/InMemoryCompiler.java | 0 .../server/checker/lal/LalComparisonTest.java | 0 .../server/checker/mal/MalComparisonTest.java | 0 .../checker/mal/MalFilterComparisonTest.java | 0 .../mal-v1-with-groovy}/pom.xml | 8 +- .../oap/meter/analyzer/Analyzer.java | 383 ++++++ .../oap/meter/analyzer/MetricConvert.java | 130 ++ .../oap/meter/analyzer/MetricRuleConfig.java | 66 + .../oap/meter/analyzer/dsl/DSL.java | 91 ++ .../meter/analyzer/dsl/DownsamplingType.java | 26 + .../EndpointEntityDescription.java | 44 + .../EntityDescription/EntityDescription.java | 28 + .../InstanceEntityDescription.java | 48 + .../ProcessEntityDescription.java | 49 + .../ProcessRelationEntityDescription.java | 49 + .../ServiceEntityDescription.java | 41 + .../ServiceRelationEntityDescription.java | 53 + .../oap/meter/analyzer/dsl/Expression.java | 152 +++ .../dsl/ExpressionParsingContext.java | 0 .../dsl/ExpressionParsingException.java | 28 + .../meter/analyzer/dsl/FilterExpression.java | 58 + .../oap/meter/analyzer/dsl/MalExpression.java | 30 + .../oap/meter/analyzer/dsl/MalFilter.java | 30 + .../oap/meter/analyzer/dsl/NumberClosure.java | 0 .../oap/meter/analyzer/dsl/Result.java | 82 ++ .../oap/meter/analyzer/dsl/Sample.java | 60 + .../oap/meter/analyzer/dsl/SampleFamily.java | 985 +++++++++++++++ .../analyzer/dsl/SampleFamilyBuilder.java | 51 + .../analyzer/dsl/SampleFamilyFunctions.java | 77 ++ .../analyzer/dsl/counter/CounterWindow.java | 88 ++ .../oap/meter/analyzer/dsl/counter/ID.java | 34 + .../dsl/registry/ProcessRegistry.java | 86 ++ .../analyzer/dsl/tagOpt/K8sRetagType.java | 52 + .../oap/meter/analyzer/dsl/tagOpt/Retag.java | 27 + .../meter/analyzer/k8s/K8sInfoRegistry.java | 161 +++ .../prometheus/PrometheusMetricConverter.java | 152 +++ .../analyzer/prometheus/rule/MetricsRule.java | 37 + .../meter/analyzer/prometheus/rule/Rule.java | 40 + .../meter/analyzer/prometheus/rule/Rules.java | 120 ++ .../oap/meter/analyzer/MetricConvertTest.java | 0 .../meter/analyzer/dsl/AggregationTest.java | 0 .../oap/meter/analyzer/dsl/AnalyzerTest.java | 0 .../meter/analyzer/dsl/ArithmeticTest.java | 0 .../oap/meter/analyzer/dsl/BasicTest.java | 0 .../oap/meter/analyzer/dsl/DecorateTest.java | 0 .../analyzer/dsl/ExpressionParsingTest.java | 0 .../oap/meter/analyzer/dsl/FilterTest.java | 0 .../oap/meter/analyzer/dsl/FunctionTest.java | 0 .../oap/meter/analyzer/dsl/IncreaseTest.java | 0 .../oap/meter/analyzer/dsl/K8sTagTest.java | 0 .../oap/meter/analyzer/dsl/ScopeTest.java | 0 .../oap/meter/analyzer/dsl/TagFilterTest.java | 0 .../meter/analyzer/dsl/ValueFilterTest.java | 0 .../dsl/counter/CounterWindowTest.java | 0 .../analyzer/dsl/rule/RuleLoaderFailTest.java | 0 .../analyzer/dsl/rule/RuleLoaderTest.java | 0 .../dsl/rule/RuleLoaderYAMLFailTest.java | 0 .../org.mockito.plugins.MockMaker | 0 .../otel-rules/illegal-yaml/test.yml | 0 .../otel-rules/single-file-case.yaml | 0 .../otel-rules/test-folder/case1.yaml | 0 .../otel-rules/test-folder/case2.yml | 0 .../otel-rules/test-folder/case3.yaml | 0 .../test-folder/deeperFolder/caseUnReach.yaml | 0 .../otel-rules/test-folder/empty.yaml | 0 .../script-compiler}/pom.xml | 21 +- 178 files changed, 14304 insertions(+), 5240 deletions(-) delete mode 100644 oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java delete mode 100644 oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java create mode 100644 oap-server/analyzer/hierarchy/CLAUDE.md create mode 100644 oap-server/analyzer/hierarchy/pom.xml create mode 100644 oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleLexer.g4 create mode 100644 oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleParser.g4 create mode 100644 oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java create mode 100644 oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleModel.java create mode 100644 oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParser.java create mode 100644 oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java create mode 100644 oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java create mode 100644 oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParserTest.java delete mode 100644 oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java delete mode 100644 oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java delete mode 100644 oap-server/analyzer/log-analyzer-v2/pom.xml delete mode 100644 oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java create mode 100644 oap-server/analyzer/log-analyzer/CLAUDE.md create mode 100644 oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALLexer.g4 create mode 100644 oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 create mode 100644 oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java create mode 100644 oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java create mode 100644 oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java create mode 100644 oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/BindingAware.java create mode 100644 oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/LalExpressionPackageHolder.java create mode 100644 oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java create mode 100644 oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java rename oap-server/analyzer/{log-analyzer-v2 => log-analyzer}/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java (100%) delete mode 100644 oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java delete mode 100644 oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java delete mode 100644 oap-server/analyzer/meter-analyzer-v2/pom.xml delete mode 100644 oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java delete mode 100644 oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java delete mode 100644 oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java create mode 100644 oap-server/analyzer/meter-analyzer/CLAUDE.md create mode 100644 oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALLexer.g4 create mode 100644 oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/rt/MalExpressionPackageHolder.java create mode 100644 oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionMetadata.java create mode 100644 oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java create mode 100644 oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParserTest.java rename oap-server/analyzer/{meter-analyzer-v2 => meter-analyzer}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java (67%) rename {oap-server/analyzer => test/script-compiler}/hierarchy-v1-v2-checker/pom.xml (84%) rename {oap-server/analyzer => test/script-compiler}/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java (100%) rename {oap-server/analyzer => test/script-compiler}/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml (100%) rename {oap-server/analyzer/hierarchy-v1 => test/script-compiler/hierarchy-v1-with-groovy}/pom.xml (94%) rename {oap-server/analyzer/hierarchy-v1 => test/script-compiler/hierarchy-v1-with-groovy}/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java (100%) rename {oap-server/analyzer/lal-transpiler => test/script-compiler/lal-v1-with-groovy}/pom.xml (82%) create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java rename {oap-server/analyzer/log-analyzer => test/script-compiler/lal-v1-with-groovy}/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java (100%) create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java rename {oap-server/analyzer/log-analyzer => test/script-compiler/lal-v1-with-groovy}/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java (100%) create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine create mode 100644 test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider rename {oap-server/analyzer/log-analyzer => test/script-compiler/lal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java (100%) rename {oap-server/analyzer/log-analyzer => test/script-compiler/lal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java (100%) rename {oap-server/analyzer/log-analyzer => test/script-compiler/lal-v1-with-groovy}/src/test/resources/log-mal-rules/placeholder.yaml (100%) rename {oap-server/analyzer/log-analyzer => test/script-compiler/lal-v1-with-groovy}/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker (100%) rename {oap-server/analyzer => test/script-compiler}/mal-lal-v1-v2-checker/pom.xml (85%) rename {oap-server/analyzer => test/script-compiler}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java (100%) rename {oap-server/analyzer => test/script-compiler}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java (100%) rename {oap-server/analyzer => test/script-compiler}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java (100%) rename {oap-server/analyzer => test/script-compiler}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java (100%) rename {oap-server/analyzer/mal-transpiler => test/script-compiler/mal-v1-with-groovy}/pom.xml (87%) create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java (100%) create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java (100%) create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java create mode 100644 test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/illegal-yaml/test.yml (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/single-file-case.yaml (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/test-folder/case1.yaml (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/test-folder/case2.yml (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/test-folder/case3.yaml (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml (100%) rename {oap-server/analyzer/meter-analyzer => test/script-compiler/mal-v1-with-groovy}/src/test/resources/otel-rules/test-folder/empty.yaml (100%) rename {oap-server/analyzer/hierarchy-v2 => test/script-compiler}/pom.xml (73%) diff --git a/docs/en/academy/groovy-replacement-plan.md b/docs/en/academy/groovy-replacement-plan.md index 101861503815..f656a7b94d69 100644 --- a/docs/en/academy/groovy-replacement-plan.md +++ b/docs/en/academy/groovy-replacement-plan.md @@ -606,6 +606,120 @@ v1 (Groovy) and v2 (Java) never coexist in the OAP runtime classpath. The `mal-l 7. **Remove `LALDelegatingScript.java`**: replaced by `LalExpression` interface 8. **Verify `server-core` has zero Groovy imports**: `HierarchyDefinitionService` and `HierarchyService` now use `BiFunction` only +### Phase 8: Replace v2 Manifest Loading with Real Compilers (ANTLR4 + Javassist) + +Phase 7 completed: v2 modules are standalone (zero Groovy), v1 depends on v2. Currently, v2 loads transpiled classes via manifest files (`META-INF/mal-expressions.txt`, `META-INF/lal-expressions.txt`) that were pre-compiled at build time. This prevents on-demand config changes since MAL/LAL/hierarchy configs are in the final package and users may want to modify them. + +The goal is to replace this manifest-based approach with **real compilers** following the OAL pattern: ANTLR4 grammar -> parser -> model -> Javassist class generation -> listener notification. This enables runtime compilation when configs change. + +#### Module Renaming: Drop `-v2` Suffix + +Since v1 modules move to `test/script-compiler/`, the v2 modules become the primary ones and lose the `-v2` suffix: +- `meter-analyzer-v2` -> `meter-analyzer` (package stays `o.a.s.oap.meter.analyzer`) +- `log-analyzer-v2` -> `log-analyzer` (package stays `o.a.s.oap.log.analyzer`) +- `hierarchy-v2` -> `hierarchy` (no v1 name conflict since v1 moves out) +- `.v2.` sub-packages (`dsl.v2.DSL`, `dsl.v2.Binding`, etc.) merge back into parent packages (`dsl.DSL`, `dsl.Binding`) + +#### Target Module Structure + +``` +oap-server/ + mal-grammar/ NEW — ANTLR4 grammar for MAL expressions + lal-grammar/ NEW — ANTLR4 grammar for LAL scripts + hierarchy-rule-grammar/ NEW — ANTLR4 grammar for hierarchy matching rules + +oap-server/analyzer/ + agent-analyzer/ (stays) + event-analyzer/ (stays) + meter-analyzer/ (renamed from meter-analyzer-v2, runtime MAL, calls mal-compiler) + log-analyzer/ (renamed from log-analyzer-v2, runtime LAL, calls lal-compiler) + hierarchy/ (renamed from hierarchy-v2, calls hierarchy-rule-compiler) + mal-compiler/ NEW — MAL expression compiler engine + lal-compiler/ NEW — LAL script compiler engine + hierarchy-rule-compiler/ NEW — hierarchy rule compiler engine + +test/script-compiler/ NEW — aggregator for v1/transpiler/checker (not in dist) + mal-groovy/ <- meter-analyzer v1 (Groovy) + lal-groovy/ <- log-analyzer v1 (Groovy) + hierarchy-groovy/ <- hierarchy-v1 (Groovy) + mal-transpiler/ <- mal-transpiler + lal-transpiler/ <- lal-transpiler + mal-lal-v1-v2-checker/ <- mal-lal-v1-v2-checker + hierarchy-v1-v2-checker/ <- hierarchy-v1-v2-checker +``` + +#### Generated Class Grouping by Config File Name + +MAL metrics come from YAML config files (e.g., `oap.yaml`, `spring-micrometer.yaml`). Each file contains multiple `metricsRules`. The compiler groups generated classes by source file name. + +- **MAL Compiler API**: `MALCompilerEngine.compile(configFileName, MetricRuleConfig)` -> grouped by file +- **Generated class naming**: `rt..MalExpr_`, e.g., `rt.oap.MalExpr_instance_jvm_cpu` +- **LAL Compiler API**: `LALCompilerEngine.compile(configFileName, List)` -> grouped by file +- **Generated class naming**: `rt..LalExpr_` + +#### Eliminate `ExpressionParsingContext` ThreadLocal (MAL only) + +The current MAL `run()` method serves dual purposes controlled by a ThreadLocal: +1. **Parse phase** (startup): `ExpressionParsingContext` ThreadLocal is set, `run()` is called with an empty map to discover which metric names the expression references +2. **Runtime phase** (every ingestion cycle): ThreadLocal is not set, `run()` computes the actual result + +This is eliminated by extracting metadata statically. The `MalExpression` interface gains a `metadata()` method: + +```java +public interface MalExpression { + /** Pure computation. No side effects. */ + SampleFamily run(Map samples); + + /** Compile-time metadata -- sample names, scope, downsampling, etc. */ + ExpressionMetadata metadata(); +} +``` + +The ANTLR4 compiler extracts all metadata from the parse tree at compile time: + +| Metadata | Extracted from | +|--|--| +| `sampleNames` | Bare identifiers (metric references) | +| `scopeType` | Terminal method: `.service()`, `.instance()`, `.endpoint()` | +| `downsampling` | Aggregation arg: `AVG`, `SUM`, `MAX`, etc. | +| `percentiles` | `.percentile()` call arguments | +| `isHistogram` | Presence of `.histogram()` in chain | + +Generated class emits metadata as static fields: + +```java +public class MalExpr_instance_jvm_cpu implements MalExpression { + private static final ExpressionMetadata METADATA = new ExpressionMetadata( + List.of("instance_jvm_cpu"), // sampleNames + ScopeType.SERVICE_INSTANCE, // from .service()/instance() call + DownsamplingType.AVG // from aggregation + ); + + @Override + public ExpressionMetadata metadata() { return METADATA; } + + @Override + public SampleFamily run(Map samples) { + return ((SampleFamily) samples.getOrDefault("instance_jvm_cpu", SampleFamily.EMPTY)) + .sum(List.of("service", "instance")); + } +} +``` + +Result: `run()` is pure computation, `metadata()` is static facts, `ExpressionParsingContext` and its ThreadLocal are deleted. No dry run with empty map at startup. + +LAL and hierarchy do **not** have this problem -- LAL passes `Binding` explicitly as a parameter, hierarchy rules are stateless lambdas. + +#### Implementation Sub-Phases + +- 8.1: Rename modules (drop -v2 suffix, flatten .v2. sub-packages) +- 8.2: Grammar modules (ANTLR4 .g4 files) +- 8.3: Compiler model + parser (no code gen) +- 8.4: Javassist code generation (including static `ExpressionMetadata` on generated MAL classes) +- 8.5: Engine integration (wire compilers into renamed modules, delete `ExpressionParsingContext`) +- 8.6: Move v1/transpiler/checker to test/script-compiler/ +- 8.7: Cleanup (remove manifests, verify zero Groovy) + --- ## 6. What Gets Removed from Runtime @@ -880,7 +994,6 @@ New MAL/LAL YAML rules added to `server-starter/src/main/resources/` are automat | LAL YAML files processed | 8 | | LAL rules transpiled | 10 (6 unique after SHA-256 dedup) | | Hierarchy rules replaced | 4 | -| Hierarchy rules replaced | 4 | | Total generated Java classes | ~1,289 | | Comparison test assertions | 1,300+ (MAL: 1,281, LAL: 19, hierarchy: 4 rules x multiple service pairs) | | Lines of transpiler code (MAL) | ~1,230 | @@ -909,4 +1022,4 @@ New MAL/LAL YAML rules added to `server-starter/src/main/resources/` are automat 4. **Phase 5**: Hierarchy v1/v2 module split -- refactor `server-core` to remove Groovy, create `hierarchy-v1/` (Groovy, checker-only) and `hierarchy-v2/` (Java lambdas, runtime) 5. **Phase 6**: Build comparison test suites -- `mal-lal-v1-v2-checker/` AND `hierarchy-v1-v2-checker/` 6. **Phase 7**: Switch `server-starter` from v1 to v2 for all three subsystems (MAL, LAL, hierarchy), remove Groovy from runtime classpath -7. **Eventually**: Remove `mal-lal-v1`, `hierarchy-v1`, and all checker modules once community confidence is established +7. **Phase 8**: Replace v2 manifest-based class loading with real ANTLR4 + Javassist compilers following the OAL pattern, enabling runtime compilation when configs change. Rename modules (drop `-v2` suffix), move v1/transpiler/checker to `test/script-compiler/`. diff --git a/oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java b/oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java deleted file mode 100644 index d85bdb99c766..000000000000 --- a/oap-server/analyzer/hierarchy-v2/src/main/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProvider.java +++ /dev/null @@ -1,86 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.server.core.config; - -import java.util.HashMap; -import java.util.Map; -import java.util.Objects; -import java.util.function.BiFunction; -import org.apache.skywalking.oap.server.core.query.type.Service; - -/** - * Pure Java hierarchy rule provider. Contains a static registry of all known - * hierarchy matching rules as Java lambdas. Zero Groovy dependency. - * - *

Rule names must match those in hierarchy-definition.yml auto-matching-rules section. - * Unknown rule names fail fast at startup with IllegalArgumentException. - */ -public final class JavaHierarchyRuleProvider implements HierarchyDefinitionService.HierarchyRuleProvider { - - private static final Map> RULE_REGISTRY; - - static { - RULE_REGISTRY = new HashMap<>(); - - // name: { (u, l) -> u.name == l.name } - RULE_REGISTRY.put("name", - (u, l) -> Objects.equals(u.getName(), l.getName())); - - // short-name: { (u, l) -> u.shortName == l.shortName } - RULE_REGISTRY.put("short-name", - (u, l) -> Objects.equals(u.getShortName(), l.getShortName())); - - // lower-short-name-remove-ns: - // { (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) - // return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); - // return false; } } - RULE_REGISTRY.put("lower-short-name-remove-ns", (u, l) -> { - final String sn = l.getShortName(); - final int dot = sn.lastIndexOf('.'); - return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); - }); - - // lower-short-name-with-fqdn: - // { (u, l) -> { if(u.shortName.lastIndexOf(':') > 0) - // return u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local'); - // return false; } } - RULE_REGISTRY.put("lower-short-name-with-fqdn", (u, l) -> { - final String sn = u.getShortName(); - final int colon = sn.lastIndexOf(':'); - return colon > 0 && Objects.equals( - sn.substring(0, colon), - l.getShortName() + ".svc.cluster.local"); - }); - } - - @Override - public Map> buildRules( - final Map ruleExpressions) { - final Map> rules = new HashMap<>(); - ruleExpressions.forEach((name, expression) -> { - final BiFunction fn = RULE_REGISTRY.get(name); - if (fn == null) { - throw new IllegalArgumentException( - "Unknown hierarchy matching rule: " + name - + ". Known rules: " + RULE_REGISTRY.keySet()); - } - rules.put(name, fn); - }); - return rules; - } -} diff --git a/oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java b/oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java deleted file mode 100644 index 699d3af4aa02..000000000000 --- a/oap-server/analyzer/hierarchy-v2/src/test/java/org/apache/skywalking/oap/server/core/config/JavaHierarchyRuleProviderTest.java +++ /dev/null @@ -1,155 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.server.core.config; - -import java.util.Map; -import java.util.function.BiFunction; -import org.apache.skywalking.oap.server.core.query.type.Service; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.junit.jupiter.api.Assertions.assertTrue; - -class JavaHierarchyRuleProviderTest { - - private JavaHierarchyRuleProvider provider; - - @BeforeEach - void setUp() { - provider = new JavaHierarchyRuleProvider(); - } - - private Service svc(final String name, final String shortName) { - final Service s = new Service(); - s.setName(name); - s.setShortName(shortName); - return s; - } - - // ---- name rule ---- - - @Test - void nameRuleMatches() { - final Map> rules = - provider.buildRules(Map.of("name", "{ (u, l) -> u.name == l.name }")); - assertTrue(rules.get("name").apply(svc("svc", "svc"), svc("svc", "svc"))); - } - - @Test - void nameRuleDoesNotMatch() { - final Map> rules = - provider.buildRules(Map.of("name", "ignored")); - assertFalse(rules.get("name").apply(svc("svc", "svc"), svc("other", "other"))); - } - - // ---- short-name rule ---- - - @Test - void shortNameRuleMatches() { - final Map> rules = - provider.buildRules(Map.of("short-name", "ignored")); - assertTrue(rules.get("short-name").apply(svc("a", "svc"), svc("b", "svc"))); - } - - @Test - void shortNameRuleDoesNotMatch() { - final Map> rules = - provider.buildRules(Map.of("short-name", "ignored")); - assertFalse(rules.get("short-name").apply(svc("a", "svc1"), svc("b", "svc2"))); - } - - // ---- lower-short-name-remove-ns rule ---- - - @Test - void lowerShortNameRemoveNsMatches() { - final Map> rules = - provider.buildRules(Map.of("lower-short-name-remove-ns", "ignored")); - // l.shortName = "svc.namespace", u.shortName = "svc" - assertTrue(rules.get("lower-short-name-remove-ns") - .apply(svc("a", "svc"), svc("b", "svc.namespace"))); - } - - @Test - void lowerShortNameRemoveNsNoDot() { - final Map> rules = - provider.buildRules(Map.of("lower-short-name-remove-ns", "ignored")); - assertFalse(rules.get("lower-short-name-remove-ns") - .apply(svc("a", "svc"), svc("b", "svc"))); - } - - @Test - void lowerShortNameRemoveNsMismatch() { - final Map> rules = - provider.buildRules(Map.of("lower-short-name-remove-ns", "ignored")); - assertFalse(rules.get("lower-short-name-remove-ns") - .apply(svc("a", "other"), svc("b", "svc.namespace"))); - } - - // ---- lower-short-name-with-fqdn rule ---- - - @Test - void lowerShortNameWithFqdnMatches() { - final Map> rules = - provider.buildRules(Map.of("lower-short-name-with-fqdn", "ignored")); - // u.shortName = "db:3306", l.shortName = "db" -> "db" == "db.svc.cluster.local"? no - // u.shortName = "db:3306", l.shortName should match: u prefix = "db", l + fqdn = "db.svc.cluster.local" - assertTrue(rules.get("lower-short-name-with-fqdn") - .apply(svc("a", "db.svc.cluster.local:3306"), svc("b", "db"))); - } - - @Test - void lowerShortNameWithFqdnNoColon() { - final Map> rules = - provider.buildRules(Map.of("lower-short-name-with-fqdn", "ignored")); - assertFalse(rules.get("lower-short-name-with-fqdn") - .apply(svc("a", "db"), svc("b", "db"))); - } - - @Test - void lowerShortNameWithFqdnWrongSuffix() { - final Map> rules = - provider.buildRules(Map.of("lower-short-name-with-fqdn", "ignored")); - assertFalse(rules.get("lower-short-name-with-fqdn") - .apply(svc("a", "db:3306"), svc("b", "other"))); - } - - // ---- unknown rule ---- - - @Test - void unknownRuleThrows() { - assertThrows(IllegalArgumentException.class, - () -> provider.buildRules(Map.of("unknown-rule", "ignored"))); - } - - // ---- builds all 4 rules ---- - - @Test - void buildsAllFourRules() { - final Map> rules = - provider.buildRules(Map.of( - "name", "ignored", - "short-name", "ignored", - "lower-short-name-remove-ns", "ignored", - "lower-short-name-with-fqdn", "ignored" - )); - assertEquals(4, rules.size()); - } -} diff --git a/oap-server/analyzer/hierarchy/CLAUDE.md b/oap-server/analyzer/hierarchy/CLAUDE.md new file mode 100644 index 000000000000..f2bf5d1d9a38 --- /dev/null +++ b/oap-server/analyzer/hierarchy/CLAUDE.md @@ -0,0 +1,113 @@ +# Hierarchy Rule Compiler + +Compiles hierarchy matching rule expressions into `BiFunction` implementation classes at runtime using ANTLR4 parsing and Javassist bytecode generation. + +## Compilation Workflow + +``` +Rule expression string (e.g., "{ (u, l) -> u.name == l.name }") + → HierarchyRuleScriptParser.parse(expression) [ANTLR4 lexer/parser → visitor] + → HierarchyRuleModel (immutable AST) + → HierarchyRuleClassGenerator.compile(ruleName, expression) + 1. classPool.makeClass() — create class implementing BiFunction + 2. generateApplyMethod(model) — emit Java source for apply(Object, Object) + 3. ctClass.toClass(HierarchyRulePackageHolder.class) — load via package anchor + → BiFunction instance +``` + +The generated class implements: +```java +Object apply(Object arg0, Object arg1) + // cast internally to Service and returns Boolean +``` + +No separate consumer/closure classes are needed — hierarchy rules are simple enough to compile into a single method body. + +## File Structure + +``` +oap-server/analyzer/hierarchy/ + src/main/antlr4/.../HierarchyRuleLexer.g4 — ANTLR4 lexer grammar + src/main/antlr4/.../HierarchyRuleParser.g4 — ANTLR4 parser grammar + + src/main/java/.../compiler/ + HierarchyRuleScriptParser.java — ANTLR4 facade: expression → AST + HierarchyRuleModel.java — Immutable AST model classes + HierarchyRuleClassGenerator.java — Javassist code generator + rt/ + HierarchyRulePackageHolder.java — Class loading anchor (empty marker) + + src/test/java/.../compiler/ + HierarchyRuleScriptParserTest.java — 5 parser tests + HierarchyRuleClassGeneratorTest.java — 4 generator tests +``` + +## Package & Class Naming + +| Component | Package / Name | +|-----------|---------------| +| Parser/Model/Generator | `org.apache.skywalking.oap.server.core.config.compiler` | +| Generated classes | `org.apache.skywalking.oap.server.core.config.compiler.rt.HierarchyRule_` | +| Package holder | `org.apache.skywalking.oap.server.core.config.compiler.rt.HierarchyRulePackageHolder` | +| Service type | `org.apache.skywalking.oap.server.core.query.type.Service` (in server-core) | + +`` is a global `AtomicInteger` counter. + +## Code Generation Details + +**Field access mapping**: Property access in expressions maps to getter methods: +- `u.name` → `u.getName()` +- `l.shortName` → `l.getShortName()` +- Generic: `x.foo` → `x.getFoo()` + +**Comparison operators**: `==` and `!=` use `java.util.Objects.equals()`. Numeric comparisons (`>`, `<`, `>=`, `<=`) emit direct operators. + +**Method chains**: `l.shortName.substring(0, l.shortName.lastIndexOf("."))` generates chained Java method calls directly. + +## Example + +**Input**: `{ (u, l) -> u.name == l.name }` + +**Generated `apply()` method**: +```java +public Object apply(Object arg0, Object arg1) { + Service u = (Service) arg0; + Service l = (Service) arg1; + return Boolean.valueOf(java.util.Objects.equals(u.getName(), l.getName())); +} +``` + +**Input with block body**: `{ (u, l) -> { if (l.shortName.lastIndexOf(".") > 0) { return u.name == l.shortName.substring(0, l.shortName.lastIndexOf(".")); } return false; } }` + +**Generated `apply()` method**: +```java +public Object apply(Object arg0, Object arg1) { + Service u = (Service) arg0; + Service l = (Service) arg1; + if (l.getShortName().lastIndexOf(".") > 0) { + return Boolean.valueOf( + java.util.Objects.equals( + u.getName(), + l.getShortName().substring(0, l.getShortName().lastIndexOf(".")))); + } + return Boolean.valueOf(false); +} +``` + +## Rule Patterns + +Four rule types are defined in `hierarchy-definition.yml`: + +| Rule Name | Expression Pattern | +|-----------|-------------------| +| `name` | `{ (u, l) -> u.name == l.name }` | +| `short-name` | `{ (u, l) -> u.shortName == l.shortName }` | +| `lower-short-name-remove-namespace` | `{ (u, l) -> { if (l.shortName.lastIndexOf(".") > 0) { return u.name == l.shortName.substring(0, l.shortName.lastIndexOf(".")); } return false; } }` | +| `lower-short-name-with-fqdn` | `{ (u, l) -> u.shortName == l.shortName.concat("." + u.shortName) }` | + +## Dependencies + +Grammar, compiler, and runtime are merged into this module: +- ANTLR4 grammar → generates lexer/parser at build time +- `server-core` — `Service` type +- `javassist` — bytecode generation diff --git a/oap-server/analyzer/hierarchy/pom.xml b/oap-server/analyzer/hierarchy/pom.xml new file mode 100644 index 000000000000..04adce9ed05b --- /dev/null +++ b/oap-server/analyzer/hierarchy/pom.xml @@ -0,0 +1,65 @@ + + + + + + analyzer + org.apache.skywalking + ${revision} + + 4.0.0 + + hierarchy + + + + org.apache.skywalking + server-core + ${project.version} + + + org.antlr + antlr4-runtime + + + org.javassist + javassist + + + + + + + org.antlr + antlr4-maven-plugin + + true + + + + antlr + + antlr4 + + + + + + + diff --git a/oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleLexer.g4 b/oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleLexer.g4 new file mode 100644 index 000000000000..2cd29899928c --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleLexer.g4 @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +// Hierarchy Rule matching expression lexer +// +// Covers expressions like: +// { (u, l) -> u.name == l.name } +// { (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); return false; } } +lexer grammar HierarchyRuleLexer; + +@Header {package org.apache.skywalking.hierarchy.rt.grammar;} + +// Keywords +IF: 'if'; +ELSE: 'else'; +RETURN: 'return'; +TRUE: 'true'; +FALSE: 'false'; + +// Comparison and logical operators +DEQ: '=='; +NEQ: '!='; +AND: '&&'; +OR: '||'; +NOT: '!'; +GT: '>'; +LT: '<'; +GTE: '>='; +LTE: '<='; + +// Delimiters +DOT: '.'; +COMMA: ','; +SEMI: ';'; +L_PAREN: '('; +R_PAREN: ')'; +L_BRACE: '{'; +R_BRACE: '}'; +ARROW: '->'; + +// Arithmetic (for substring index arguments) +PLUS: '+'; +MINUS: '-'; + +// Literals +NUMBER + : Digit+ + ; + +STRING + : '\'' (~['\\\r\n] | EscapeSequence)* '\'' + | '"' (~["\\\r\n] | EscapeSequence)* '"' + ; + +// Comments +LINE_COMMENT + : '//' ~[\r\n]* -> channel(HIDDEN) + ; + +BLOCK_COMMENT + : '/*' .*? '*/' -> channel(HIDDEN) + ; + +// Whitespace +WS + : [ \t\r\n]+ -> channel(HIDDEN) + ; + +// Identifiers +IDENTIFIER + : Letter LetterOrDigit* + ; + +// Fragments +fragment EscapeSequence + : '\\' [btnfr"'\\] + ; + +fragment Digit + : [0-9] + ; + +fragment Letter + : [a-zA-Z_] + ; + +fragment LetterOrDigit + : Letter + | [0-9] + ; diff --git a/oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleParser.g4 b/oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleParser.g4 new file mode 100644 index 000000000000..7d30c016c1f9 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/antlr4/org/apache/skywalking/hierarchy/rt/grammar/HierarchyRuleParser.g4 @@ -0,0 +1,135 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +// Hierarchy Rule matching expression parser +// +// Parses expressions from hierarchy-definition.yml auto-matching-rules: +// name: "{ (u, l) -> u.name == l.name }" +// short-name: "{ (u, l) -> u.shortName == l.shortName }" +// lower-short-name-remove-ns: +// "{ (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); return false; } }" +// lower-short-name-with-fqdn: +// "{ (u, l) -> { if(u.shortName.lastIndexOf(':') > 0) return u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local'); return false; } }" +parser grammar HierarchyRuleParser; + +@Header {package org.apache.skywalking.hierarchy.rt.grammar;} + +options { tokenVocab=HierarchyRuleLexer; } + +// ==================== Top-level ==================== + +// { (u, l) -> body } +matchingRule + : L_BRACE L_PAREN param COMMA param R_PAREN ARROW ruleBody R_BRACE EOF + ; + +param + : IDENTIFIER + ; + +ruleBody + : simpleExpression // u.name == l.name + | blockBody // { if(...) ...; return false; } + ; + +// ==================== Block body ==================== + +blockBody + : L_BRACE statement+ R_BRACE + ; + +statement + : ifStatement + | returnStatement + ; + +ifStatement + : IF L_PAREN condition R_PAREN + (returnStatement | blockBody) + (ELSE IF L_PAREN condition R_PAREN + (returnStatement | blockBody) + )* + (ELSE + (returnStatement | blockBody) + )? + ; + +returnStatement + : RETURN returnValue SEMI? + ; + +returnValue + : ruleExpr DEQ ruleExpr # returnComparison + | ruleExpr NEQ ruleExpr # returnNeqComparison + | ruleExpr # returnExpr + ; + +// ==================== Conditions ==================== + +condition + : condition AND condition # condAnd + | condition OR condition # condOr + | NOT condition # condNot + | L_PAREN condition R_PAREN # condParen + | ruleExpr DEQ ruleExpr # condEq + | ruleExpr NEQ ruleExpr # condNeq + | ruleExpr GT ruleExpr # condGt + | ruleExpr LT ruleExpr # condLt + | ruleExpr GTE ruleExpr # condGte + | ruleExpr LTE ruleExpr # condLte + | ruleExpr # condExpr + ; + +// ==================== Expressions ==================== + +simpleExpression + : ruleExpr DEQ ruleExpr + | ruleExpr NEQ ruleExpr + ; + +ruleExpr + : ruleExpr PLUS ruleExpr # exprAdd + | ruleExpr MINUS ruleExpr # exprSub + | ruleExprPrimary # exprPrimary + ; + +ruleExprPrimary + : methodChain # exprMethodChain + | STRING # exprString + | NUMBER # exprNumber + | TRUE # exprTrue + | FALSE # exprFalse + ; + +// ==================== Method chains ==================== + +// u.name, l.shortName, l.shortName.lastIndexOf('.'), +// u.shortName.substring(0, l.shortName.lastIndexOf(':')) +// l.shortName.concat('.svc.cluster.local') +methodChain + : IDENTIFIER (DOT chainSegment)+ + ; + +chainSegment + : IDENTIFIER L_PAREN argList? R_PAREN # chainMethodCall + | IDENTIFIER # chainFieldAccess + ; + +argList + : ruleExpr (COMMA ruleExpr)* + ; diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java new file mode 100644 index 000000000000..754bffa27881 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java @@ -0,0 +1,294 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler; + +import java.util.List; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.function.BiFunction; +import javassist.ClassPool; +import javassist.CtClass; +import javassist.CtNewConstructor; +import javassist.CtNewMethod; +import org.apache.skywalking.oap.server.core.config.compiler.rt.HierarchyRulePackageHolder; +import org.apache.skywalking.oap.server.core.query.type.Service; + +/** + * Generates {@link BiFunction BiFunction<Service, Service, Boolean>} implementation classes + * from {@link HierarchyRuleModel} AST using Javassist bytecode generation. + */ +public final class HierarchyRuleClassGenerator { + + private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(0); + + private static final String PACKAGE_PREFIX = + "org.apache.skywalking.oap.server.core.config.compiler.rt."; + + private final ClassPool classPool; + + public HierarchyRuleClassGenerator() { + this(ClassPool.getDefault()); + } + + public HierarchyRuleClassGenerator(final ClassPool classPool) { + this.classPool = classPool; + } + + /** + * Compiles a hierarchy rule expression into a BiFunction class. + * + * @param ruleName the rule name (e.g., "name", "short-name") + * @param expression the rule expression string + * @return a BiFunction that matches two Service objects + */ + @SuppressWarnings("unchecked") + public BiFunction compile( + final String ruleName, final String expression) throws Exception { + final HierarchyRuleModel model = HierarchyRuleScriptParser.parse(expression); + final String className = PACKAGE_PREFIX + "HierarchyRule_" + + CLASS_COUNTER.getAndIncrement(); + + final CtClass ctClass = classPool.makeClass(className); + ctClass.addInterface(classPool.get("java.util.function.BiFunction")); + + ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); + + final String applyBody = generateApplyMethod(model); + ctClass.addMethod(CtNewMethod.make(applyBody, ctClass)); + + final Class clazz = ctClass.toClass(HierarchyRulePackageHolder.class); + ctClass.detach(); + return (BiFunction) clazz.getDeclaredConstructor().newInstance(); + } + + private String generateApplyMethod(final HierarchyRuleModel model) { + final StringBuilder sb = new StringBuilder(); + sb.append("public Object apply(Object arg0, Object arg1) {\n"); + sb.append(" org.apache.skywalking.oap.server.core.query.type.Service "); + sb.append(model.getUpperParam()).append(" = (org.apache.skywalking.oap.server.core.query.type.Service) arg0;\n"); + sb.append(" org.apache.skywalking.oap.server.core.query.type.Service "); + sb.append(model.getLowerParam()).append(" = (org.apache.skywalking.oap.server.core.query.type.Service) arg1;\n"); + + generateRuleBody(sb, model.getBody()); + + sb.append("}\n"); + return sb.toString(); + } + + private void generateRuleBody(final StringBuilder sb, + final HierarchyRuleModel.RuleBody body) { + if (body instanceof HierarchyRuleModel.SimpleComparison) { + final HierarchyRuleModel.SimpleComparison cmp = + (HierarchyRuleModel.SimpleComparison) body; + sb.append(" return Boolean.valueOf("); + generateComparison(sb, cmp.getLeft(), cmp.getOp(), cmp.getRight()); + sb.append(");\n"); + } else if (body instanceof HierarchyRuleModel.BlockBody) { + final HierarchyRuleModel.BlockBody block = (HierarchyRuleModel.BlockBody) body; + for (final HierarchyRuleModel.Statement stmt : block.getStatements()) { + generateStatement(sb, stmt); + } + } + } + + private void generateStatement(final StringBuilder sb, + final HierarchyRuleModel.Statement stmt) { + if (stmt instanceof HierarchyRuleModel.IfStatement) { + generateIfStatement(sb, (HierarchyRuleModel.IfStatement) stmt); + } else if (stmt instanceof HierarchyRuleModel.ReturnStatement) { + generateReturnStatement(sb, (HierarchyRuleModel.ReturnStatement) stmt); + } + } + + private void generateIfStatement(final StringBuilder sb, + final HierarchyRuleModel.IfStatement ifStmt) { + sb.append(" if ("); + generateCondition(sb, ifStmt.getCondition()); + sb.append(") {\n"); + for (final HierarchyRuleModel.Statement s : ifStmt.getThenBranch()) { + generateStatement(sb, s); + } + sb.append(" }\n"); + if (ifStmt.getElseBranch() != null && !ifStmt.getElseBranch().isEmpty()) { + sb.append(" else {\n"); + for (final HierarchyRuleModel.Statement s : ifStmt.getElseBranch()) { + generateStatement(sb, s); + } + sb.append(" }\n"); + } + } + + private void generateReturnStatement(final StringBuilder sb, + final HierarchyRuleModel.ReturnStatement retStmt) { + final HierarchyRuleModel.Expr expr = retStmt.getValue(); + if (expr instanceof HierarchyRuleModel.SimpleComparison) { + final HierarchyRuleModel.SimpleComparison cmp = + (HierarchyRuleModel.SimpleComparison) expr; + sb.append(" return Boolean.valueOf("); + generateComparison(sb, cmp.getLeft(), cmp.getOp(), cmp.getRight()); + sb.append(");\n"); + } else if (expr instanceof HierarchyRuleModel.BoolLiteralExpr) { + sb.append(" return Boolean.valueOf(") + .append(((HierarchyRuleModel.BoolLiteralExpr) expr).isValue()) + .append(");\n"); + } else { + sb.append(" return Boolean.valueOf("); + generateExpr(sb, expr); + sb.append(" != null);\n"); + } + } + + private void generateComparison(final StringBuilder sb, + final HierarchyRuleModel.Expr left, + final HierarchyRuleModel.CompareOp op, + final HierarchyRuleModel.Expr right) { + switch (op) { + case EQ: + sb.append("java.util.Objects.equals("); + generateExpr(sb, left); + sb.append(", "); + generateExpr(sb, right); + sb.append(")"); + break; + case NEQ: + sb.append("!java.util.Objects.equals("); + generateExpr(sb, left); + sb.append(", "); + generateExpr(sb, right); + sb.append(")"); + break; + case GT: + generateExpr(sb, left); + sb.append(" > "); + generateExpr(sb, right); + break; + case LT: + generateExpr(sb, left); + sb.append(" < "); + generateExpr(sb, right); + break; + case GTE: + generateExpr(sb, left); + sb.append(" >= "); + generateExpr(sb, right); + break; + case LTE: + generateExpr(sb, left); + sb.append(" <= "); + generateExpr(sb, right); + break; + default: + throw new IllegalArgumentException("Unsupported comparison op: " + op); + } + } + + private void generateCondition(final StringBuilder sb, + final HierarchyRuleModel.Condition cond) { + if (cond instanceof HierarchyRuleModel.ComparisonCondition) { + final HierarchyRuleModel.ComparisonCondition cc = + (HierarchyRuleModel.ComparisonCondition) cond; + generateComparison(sb, cc.getLeft(), cc.getOp(), cc.getRight()); + } else if (cond instanceof HierarchyRuleModel.LogicalCondition) { + final HierarchyRuleModel.LogicalCondition lc = + (HierarchyRuleModel.LogicalCondition) cond; + sb.append("("); + generateCondition(sb, lc.getLeft()); + sb.append(lc.getOp() == HierarchyRuleModel.LogicalOp.AND ? " && " : " || "); + generateCondition(sb, lc.getRight()); + sb.append(")"); + } else if (cond instanceof HierarchyRuleModel.NotCondition) { + sb.append("!("); + generateCondition(sb, ((HierarchyRuleModel.NotCondition) cond).getInner()); + sb.append(")"); + } else if (cond instanceof HierarchyRuleModel.ExprCondition) { + generateExpr(sb, ((HierarchyRuleModel.ExprCondition) cond).getExpr()); + } + } + + private void generateExpr(final StringBuilder sb, + final HierarchyRuleModel.Expr expr) { + if (expr instanceof HierarchyRuleModel.MethodChainExpr) { + generateMethodChainExpr(sb, (HierarchyRuleModel.MethodChainExpr) expr); + } else if (expr instanceof HierarchyRuleModel.StringLiteralExpr) { + sb.append('"') + .append(escapeJava(((HierarchyRuleModel.StringLiteralExpr) expr).getValue())) + .append('"'); + } else if (expr instanceof HierarchyRuleModel.NumberLiteralExpr) { + sb.append(((HierarchyRuleModel.NumberLiteralExpr) expr).getValue()); + } else if (expr instanceof HierarchyRuleModel.BoolLiteralExpr) { + sb.append(((HierarchyRuleModel.BoolLiteralExpr) expr).isValue()); + } else if (expr instanceof HierarchyRuleModel.BinaryExpr) { + final HierarchyRuleModel.BinaryExpr bin = (HierarchyRuleModel.BinaryExpr) expr; + generateExpr(sb, bin.getLeft()); + sb.append(bin.getOp() == HierarchyRuleModel.ArithmeticOp.ADD ? " + " : " - "); + generateExpr(sb, bin.getRight()); + } else if (expr instanceof HierarchyRuleModel.SimpleComparison) { + final HierarchyRuleModel.SimpleComparison cmp = + (HierarchyRuleModel.SimpleComparison) expr; + generateComparison(sb, cmp.getLeft(), cmp.getOp(), cmp.getRight()); + } + } + + private void generateMethodChainExpr(final StringBuilder sb, + final HierarchyRuleModel.MethodChainExpr expr) { + sb.append(expr.getTarget()); + for (final HierarchyRuleModel.ChainSegment seg : expr.getSegments()) { + sb.append('.'); + if (seg instanceof HierarchyRuleModel.FieldAccess) { + final String fieldName = ((HierarchyRuleModel.FieldAccess) seg).getName(); + sb.append(toGetter(fieldName)).append("()"); + } else if (seg instanceof HierarchyRuleModel.MethodCallSegment) { + final HierarchyRuleModel.MethodCallSegment mc = + (HierarchyRuleModel.MethodCallSegment) seg; + sb.append(mc.getName()).append('('); + final List args = mc.getArguments(); + for (int i = 0; i < args.size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateExpr(sb, args.get(i)); + } + sb.append(')'); + } + } + } + + private static String toGetter(final String fieldName) { + if ("name".equals(fieldName)) { + return "getName"; + } else if ("shortName".equals(fieldName)) { + return "getShortName"; + } + return "get" + Character.toUpperCase(fieldName.charAt(0)) + fieldName.substring(1); + } + + private static String escapeJava(final String s) { + return s.replace("\\", "\\\\") + .replace("\"", "\\\"") + .replace("\n", "\\n") + .replace("\r", "\\r") + .replace("\t", "\\t"); + } + + /** + * Generates the Java source body of the apply method for debugging/testing. + */ + public String generateSource(final String expression) { + final HierarchyRuleModel model = HierarchyRuleScriptParser.parse(expression); + return generateApplyMethod(model); + } +} diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleModel.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleModel.java new file mode 100644 index 000000000000..d9ecc4d70700 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleModel.java @@ -0,0 +1,260 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler; + +import java.util.Collections; +import java.util.List; +import lombok.Getter; + +/** + * Immutable AST model for hierarchy matching rule expressions. + * Represents parsed expressions like: + * 

+ *   { (u, l) -> u.name == l.name }
+ *   { (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) return ...; return false; } }
+ *
+ */ +@Getter +public final class HierarchyRuleModel { + private final String upperParam; + private final String lowerParam; + private final RuleBody body; + + private HierarchyRuleModel(final String upperParam, final String lowerParam, final RuleBody body) { + this.upperParam = upperParam; + this.lowerParam = lowerParam; + this.body = body; + } + + public static HierarchyRuleModel of(final String upperParam, final String lowerParam, final RuleBody body) { + return new HierarchyRuleModel(upperParam, lowerParam, body); + } + + /** + * Rule body — either a simple comparison or a block with if/return statements. + */ + public interface RuleBody { + } + + /** + * Simple comparison body: {@code u.name == l.name} + */ + @Getter + public static final class SimpleComparison implements RuleBody, Expr { + private final Expr left; + private final CompareOp op; + private final Expr right; + + public SimpleComparison(final Expr left, final CompareOp op, final Expr right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + /** + * Block body with multiple statements: {@code { if(...) return ...; return false; }} + */ + @Getter + public static final class BlockBody implements RuleBody { + private final List statements; + + public BlockBody(final List statements) { + this.statements = Collections.unmodifiableList(statements); + } + } + + // ==================== Statements ==================== + + public interface Statement { + } + + @Getter + public static final class IfStatement implements Statement { + private final Condition condition; + private final List thenBranch; + private final List elseBranch; + + public IfStatement(final Condition condition, + final List thenBranch, + final List elseBranch) { + this.condition = condition; + this.thenBranch = Collections.unmodifiableList(thenBranch); + this.elseBranch = elseBranch != null + ? Collections.unmodifiableList(elseBranch) : Collections.emptyList(); + } + } + + @Getter + public static final class ReturnStatement implements Statement { + private final Expr value; + + public ReturnStatement(final Expr value) { + this.value = value; + } + } + + // ==================== Conditions ==================== + + public interface Condition { + } + + @Getter + public static final class ComparisonCondition implements Condition { + private final Expr left; + private final CompareOp op; + private final Expr right; + + public ComparisonCondition(final Expr left, final CompareOp op, final Expr right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + @Getter + public static final class LogicalCondition implements Condition { + private final Condition left; + private final LogicalOp op; + private final Condition right; + + public LogicalCondition(final Condition left, final LogicalOp op, final Condition right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + @Getter + public static final class NotCondition implements Condition { + private final Condition inner; + + public NotCondition(final Condition inner) { + this.inner = inner; + } + } + + @Getter + public static final class ExprCondition implements Condition { + private final Expr expr; + + public ExprCondition(final Expr expr) { + this.expr = expr; + } + } + + // ==================== Expressions ==================== + + public interface Expr { + } + + /** + * Method chain: {@code u.name}, {@code l.shortName.lastIndexOf('.')}, + * {@code u.shortName.substring(0, l.shortName.lastIndexOf(':'))} + */ + @Getter + public static final class MethodChainExpr implements Expr { + private final String target; + private final List segments; + + public MethodChainExpr(final String target, final List segments) { + this.target = target; + this.segments = Collections.unmodifiableList(segments); + } + } + + @Getter + public static final class StringLiteralExpr implements Expr { + private final String value; + + public StringLiteralExpr(final String value) { + this.value = value; + } + } + + @Getter + public static final class NumberLiteralExpr implements Expr { + private final long value; + + public NumberLiteralExpr(final long value) { + this.value = value; + } + } + + @Getter + public static final class BoolLiteralExpr implements Expr { + private final boolean value; + + public BoolLiteralExpr(final boolean value) { + this.value = value; + } + } + + @Getter + public static final class BinaryExpr implements Expr { + private final Expr left; + private final ArithmeticOp op; + private final Expr right; + + public BinaryExpr(final Expr left, final ArithmeticOp op, final Expr right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + // ==================== Chain segments ==================== + + public interface ChainSegment { + String getName(); + } + + @Getter + public static final class FieldAccess implements ChainSegment { + private final String name; + + public FieldAccess(final String name) { + this.name = name; + } + } + + @Getter + public static final class MethodCallSegment implements ChainSegment { + private final String name; + private final List arguments; + + public MethodCallSegment(final String name, final List arguments) { + this.name = name; + this.arguments = Collections.unmodifiableList(arguments); + } + } + + // ==================== Enums ==================== + + public enum CompareOp { + EQ, NEQ, GT, LT, GTE, LTE + } + + public enum LogicalOp { + AND, OR + } + + public enum ArithmeticOp { + ADD, SUB + } +} diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParser.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParser.java new file mode 100644 index 000000000000..84dd3db2e200 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParser.java @@ -0,0 +1,375 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler; + +import java.util.ArrayList; +import java.util.List; +import org.antlr.v4.runtime.BaseErrorListener; +import org.antlr.v4.runtime.CharStreams; +import org.antlr.v4.runtime.CommonTokenStream; +import org.antlr.v4.runtime.RecognitionException; +import org.antlr.v4.runtime.Recognizer; +import org.apache.skywalking.hierarchy.rt.grammar.HierarchyRuleLexer; +import org.apache.skywalking.hierarchy.rt.grammar.HierarchyRuleParser; +import org.apache.skywalking.hierarchy.rt.grammar.HierarchyRuleParserBaseVisitor; + +/** + * Facade: parses hierarchy rule expression strings into {@link HierarchyRuleModel}. + * + * 
+ *   HierarchyRuleModel model = HierarchyRuleScriptParser.parse(
+ *       "{ (u, l) -> u.name == l.name }");
+ *
+ */ +public final class HierarchyRuleScriptParser { + + private HierarchyRuleScriptParser() { + } + + public static HierarchyRuleModel parse(final String expression) { + final HierarchyRuleLexer lexer = new HierarchyRuleLexer( + CharStreams.fromString(expression)); + final CommonTokenStream tokens = new CommonTokenStream(lexer); + final HierarchyRuleParser parser = new HierarchyRuleParser(tokens); + + final List errors = new ArrayList<>(); + parser.removeErrorListeners(); + parser.addErrorListener(new BaseErrorListener() { + @Override + public void syntaxError(final Recognizer recognizer, + final Object offendingSymbol, + final int line, + final int charPositionInLine, + final String msg, + final RecognitionException e) { + errors.add(line + ":" + charPositionInLine + " " + msg); + } + }); + + final HierarchyRuleParser.MatchingRuleContext tree = parser.matchingRule(); + if (!errors.isEmpty()) { + throw new IllegalArgumentException( + "Hierarchy rule parsing failed: " + String.join("; ", errors) + + " in expression: " + expression); + } + + return new HierarchyRuleModelVisitor().visit(tree); + } + + /** + * Visitor that transforms the ANTLR4 parse tree into {@link HierarchyRuleModel}. + */ + private static final class HierarchyRuleModelVisitor + extends HierarchyRuleParserBaseVisitor { + + @Override + public HierarchyRuleModel visitMatchingRule(final HierarchyRuleParser.MatchingRuleContext ctx) { + final String upperParam = ctx.param(0).getText(); + final String lowerParam = ctx.param(1).getText(); + final HierarchyRuleModel.RuleBody body = convertRuleBody(ctx.ruleBody()); + return HierarchyRuleModel.of(upperParam, lowerParam, body); + } + + private HierarchyRuleModel.RuleBody convertRuleBody( + final HierarchyRuleParser.RuleBodyContext ctx) { + if (ctx.simpleExpression() != null) { + return convertSimpleExpression(ctx.simpleExpression()); + } + return convertBlockBody(ctx.blockBody()); + } + + private HierarchyRuleModel.SimpleComparison convertSimpleExpression( + final HierarchyRuleParser.SimpleExpressionContext ctx) { + final HierarchyRuleModel.Expr left = new ExprVisitor().visitRuleExpr(ctx.ruleExpr(0)); + final HierarchyRuleModel.Expr right = new ExprVisitor().visitRuleExpr(ctx.ruleExpr(1)); + final HierarchyRuleModel.CompareOp op = ctx.DEQ() != null + ? HierarchyRuleModel.CompareOp.EQ : HierarchyRuleModel.CompareOp.NEQ; + return new HierarchyRuleModel.SimpleComparison(left, op, right); + } + + private HierarchyRuleModel.BlockBody convertBlockBody( + final HierarchyRuleParser.BlockBodyContext ctx) { + final List stmts = new ArrayList<>(); + for (final HierarchyRuleParser.StatementContext stmtCtx : ctx.statement()) { + stmts.add(convertStatement(stmtCtx)); + } + return new HierarchyRuleModel.BlockBody(stmts); + } + + private HierarchyRuleModel.Statement convertStatement( + final HierarchyRuleParser.StatementContext ctx) { + if (ctx.ifStatement() != null) { + return convertIfStatement(ctx.ifStatement()); + } + return convertReturnStatement(ctx.returnStatement()); + } + + private HierarchyRuleModel.IfStatement convertIfStatement( + final HierarchyRuleParser.IfStatementContext ctx) { + final HierarchyRuleModel.Condition condition = + new ConditionVisitor().visit(ctx.condition(0)); + + final List thenBranch = new ArrayList<>(); + if (ctx.returnStatement(0) != null) { + thenBranch.add(convertReturnStatement(ctx.returnStatement(0))); + } else if (ctx.blockBody(0) != null) { + thenBranch.addAll(convertBlockBody(ctx.blockBody(0)).getStatements()); + } + + final List elseBranch = new ArrayList<>(); + // Handle else-if and else branches + final int condCount = ctx.condition().size(); + final int retCount = ctx.returnStatement().size(); + final int blockCount = ctx.blockBody().size(); + + // If there are more conditions (else if branches) + if (condCount > 1 || retCount > 1 || blockCount > 1) { + // Simplification: flatten else-if into the else branch + // For the current hierarchy rules, we don't have else-if patterns + // so this handles the basic else case + if (retCount > 1) { + elseBranch.add(convertReturnStatement(ctx.returnStatement(retCount - 1))); + } else if (blockCount > 1) { + elseBranch.addAll( + convertBlockBody(ctx.blockBody(blockCount - 1)).getStatements()); + } + } + + return new HierarchyRuleModel.IfStatement(condition, thenBranch, elseBranch); + } + + private HierarchyRuleModel.ReturnStatement convertReturnStatement( + final HierarchyRuleParser.ReturnStatementContext ctx) { + final HierarchyRuleParser.ReturnValueContext rv = ctx.returnValue(); + if (rv instanceof HierarchyRuleParser.ReturnComparisonContext) { + final HierarchyRuleParser.ReturnComparisonContext rc = + (HierarchyRuleParser.ReturnComparisonContext) rv; + final ExprVisitor ev = new ExprVisitor(); + final HierarchyRuleModel.SimpleComparison comp = + new HierarchyRuleModel.SimpleComparison( + ev.visitRuleExpr(rc.ruleExpr(0)), + HierarchyRuleModel.CompareOp.EQ, + ev.visitRuleExpr(rc.ruleExpr(1))); + return new HierarchyRuleModel.ReturnStatement(comp); + } + if (rv instanceof HierarchyRuleParser.ReturnNeqComparisonContext) { + final HierarchyRuleParser.ReturnNeqComparisonContext rnc = + (HierarchyRuleParser.ReturnNeqComparisonContext) rv; + final ExprVisitor ev = new ExprVisitor(); + final HierarchyRuleModel.SimpleComparison comp = + new HierarchyRuleModel.SimpleComparison( + ev.visitRuleExpr(rnc.ruleExpr(0)), + HierarchyRuleModel.CompareOp.NEQ, + ev.visitRuleExpr(rnc.ruleExpr(1))); + return new HierarchyRuleModel.ReturnStatement(comp); + } + // returnExpr + final HierarchyRuleParser.ReturnExprContext re = + (HierarchyRuleParser.ReturnExprContext) rv; + final HierarchyRuleModel.Expr value = new ExprVisitor().visitRuleExpr(re.ruleExpr()); + return new HierarchyRuleModel.ReturnStatement(value); + } + } + + /** + * Visitor for condition nodes. + */ + private static final class ConditionVisitor + extends HierarchyRuleParserBaseVisitor { + + @Override + public HierarchyRuleModel.Condition visitCondAnd( + final HierarchyRuleParser.CondAndContext ctx) { + return new HierarchyRuleModel.LogicalCondition( + visit(ctx.condition(0)), + HierarchyRuleModel.LogicalOp.AND, + visit(ctx.condition(1))); + } + + @Override + public HierarchyRuleModel.Condition visitCondOr( + final HierarchyRuleParser.CondOrContext ctx) { + return new HierarchyRuleModel.LogicalCondition( + visit(ctx.condition(0)), + HierarchyRuleModel.LogicalOp.OR, + visit(ctx.condition(1))); + } + + @Override + public HierarchyRuleModel.Condition visitCondNot( + final HierarchyRuleParser.CondNotContext ctx) { + return new HierarchyRuleModel.NotCondition(visit(ctx.condition())); + } + + @Override + public HierarchyRuleModel.Condition visitCondParen( + final HierarchyRuleParser.CondParenContext ctx) { + return visit(ctx.condition()); + } + + @Override + public HierarchyRuleModel.Condition visitCondEq( + final HierarchyRuleParser.CondEqContext ctx) { + final ExprVisitor ev = new ExprVisitor(); + return new HierarchyRuleModel.ComparisonCondition( + ev.visitRuleExpr(ctx.ruleExpr(0)), + HierarchyRuleModel.CompareOp.EQ, + ev.visitRuleExpr(ctx.ruleExpr(1))); + } + + @Override + public HierarchyRuleModel.Condition visitCondNeq( + final HierarchyRuleParser.CondNeqContext ctx) { + final ExprVisitor ev = new ExprVisitor(); + return new HierarchyRuleModel.ComparisonCondition( + ev.visitRuleExpr(ctx.ruleExpr(0)), + HierarchyRuleModel.CompareOp.NEQ, + ev.visitRuleExpr(ctx.ruleExpr(1))); + } + + @Override + public HierarchyRuleModel.Condition visitCondGt( + final HierarchyRuleParser.CondGtContext ctx) { + final ExprVisitor ev = new ExprVisitor(); + return new HierarchyRuleModel.ComparisonCondition( + ev.visitRuleExpr(ctx.ruleExpr(0)), + HierarchyRuleModel.CompareOp.GT, + ev.visitRuleExpr(ctx.ruleExpr(1))); + } + + @Override + public HierarchyRuleModel.Condition visitCondLt( + final HierarchyRuleParser.CondLtContext ctx) { + final ExprVisitor ev = new ExprVisitor(); + return new HierarchyRuleModel.ComparisonCondition( + ev.visitRuleExpr(ctx.ruleExpr(0)), + HierarchyRuleModel.CompareOp.LT, + ev.visitRuleExpr(ctx.ruleExpr(1))); + } + + @Override + public HierarchyRuleModel.Condition visitCondExpr( + final HierarchyRuleParser.CondExprContext ctx) { + final ExprVisitor ev = new ExprVisitor(); + return new HierarchyRuleModel.ExprCondition(ev.visitRuleExpr(ctx.ruleExpr())); + } + } + + /** + * Visitor for expression nodes. + */ + private static final class ExprVisitor + extends HierarchyRuleParserBaseVisitor { + + public HierarchyRuleModel.Expr visitRuleExpr( + final HierarchyRuleParser.RuleExprContext ctx) { + return visit(ctx); + } + + @Override + public HierarchyRuleModel.Expr visitExprAdd( + final HierarchyRuleParser.ExprAddContext ctx) { + return new HierarchyRuleModel.BinaryExpr( + visit(ctx.ruleExpr(0)), + HierarchyRuleModel.ArithmeticOp.ADD, + visit(ctx.ruleExpr(1))); + } + + @Override + public HierarchyRuleModel.Expr visitExprSub( + final HierarchyRuleParser.ExprSubContext ctx) { + return new HierarchyRuleModel.BinaryExpr( + visit(ctx.ruleExpr(0)), + HierarchyRuleModel.ArithmeticOp.SUB, + visit(ctx.ruleExpr(1))); + } + + @Override + public HierarchyRuleModel.Expr visitExprPrimary( + final HierarchyRuleParser.ExprPrimaryContext ctx) { + return visit(ctx.ruleExprPrimary()); + } + + @Override + public HierarchyRuleModel.Expr visitExprMethodChain( + final HierarchyRuleParser.ExprMethodChainContext ctx) { + return convertMethodChain(ctx.methodChain()); + } + + @Override + public HierarchyRuleModel.Expr visitExprString( + final HierarchyRuleParser.ExprStringContext ctx) { + return new HierarchyRuleModel.StringLiteralExpr(stripQuotes(ctx.STRING().getText())); + } + + @Override + public HierarchyRuleModel.Expr visitExprNumber( + final HierarchyRuleParser.ExprNumberContext ctx) { + return new HierarchyRuleModel.NumberLiteralExpr(Long.parseLong(ctx.NUMBER().getText())); + } + + @Override + public HierarchyRuleModel.Expr visitExprTrue( + final HierarchyRuleParser.ExprTrueContext ctx) { + return new HierarchyRuleModel.BoolLiteralExpr(true); + } + + @Override + public HierarchyRuleModel.Expr visitExprFalse( + final HierarchyRuleParser.ExprFalseContext ctx) { + return new HierarchyRuleModel.BoolLiteralExpr(false); + } + + private HierarchyRuleModel.MethodChainExpr convertMethodChain( + final HierarchyRuleParser.MethodChainContext ctx) { + final String target = ctx.IDENTIFIER().getText(); + final List segments = new ArrayList<>(); + for (final HierarchyRuleParser.ChainSegmentContext seg : ctx.chainSegment()) { + segments.add(convertChainSegment(seg)); + } + return new HierarchyRuleModel.MethodChainExpr(target, segments); + } + + private HierarchyRuleModel.ChainSegment convertChainSegment( + final HierarchyRuleParser.ChainSegmentContext ctx) { + if (ctx instanceof HierarchyRuleParser.ChainMethodCallContext) { + final HierarchyRuleParser.ChainMethodCallContext mc = + (HierarchyRuleParser.ChainMethodCallContext) ctx; + final String name = mc.IDENTIFIER().getText(); + final List args = new ArrayList<>(); + if (mc.argList() != null) { + for (final HierarchyRuleParser.RuleExprContext argCtx : + mc.argList().ruleExpr()) { + args.add(visit(argCtx)); + } + } + return new HierarchyRuleModel.MethodCallSegment(name, args); + } + final HierarchyRuleParser.ChainFieldAccessContext fa = + (HierarchyRuleParser.ChainFieldAccessContext) ctx; + return new HierarchyRuleModel.FieldAccess(fa.IDENTIFIER().getText()); + } + } + + private static String stripQuotes(final String s) { + if (s.length() >= 2 && (s.charAt(0) == '\'' || s.charAt(0) == '"')) { + return s.substring(1, s.length() - 1); + } + return s; + } +} diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java new file mode 100644 index 000000000000..31d74d245d67 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler.rt; + +/** + * Empty marker class used as the class loading anchor for Javassist + * {@code CtClass.toClass(Class)} on JDK 16+. + * Generated hierarchy rule classes are loaded in this package. + */ +public class HierarchyRulePackageHolder { +} diff --git a/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java b/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java new file mode 100644 index 000000000000..1054bcb27bb0 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java @@ -0,0 +1,122 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler; + +import java.util.function.BiFunction; +import javassist.ClassPool; +import org.apache.skywalking.oap.server.core.query.type.Service; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class HierarchyRuleClassGeneratorTest { + + private HierarchyRuleClassGenerator generator; + + @BeforeEach + void setUp() { + generator = new HierarchyRuleClassGenerator(new ClassPool(true)); + } + + @Test + void compileSimpleNameEquality() throws Exception { + final BiFunction fn = generator.compile( + "name", "{ (u, l) -> u.name == l.name }"); + + assertNotNull(fn); + + final Service upper = new Service(); + upper.setName("svc-a"); + final Service lower = new Service(); + lower.setName("svc-a"); + assertTrue(fn.apply(upper, lower)); + + lower.setName("svc-b"); + assertFalse(fn.apply(upper, lower)); + } + + @Test + void compileShortNameEquality() throws Exception { + final BiFunction fn = generator.compile( + "short-name", "{ (u, l) -> u.shortName == l.shortName }"); + + assertNotNull(fn); + + final Service upper = new Service(); + upper.setShortName("svc"); + final Service lower = new Service(); + lower.setShortName("svc"); + assertTrue(fn.apply(upper, lower)); + + lower.setShortName("other"); + assertFalse(fn.apply(upper, lower)); + } + + @Test + void compileLowerShortNameRemoveNs() throws Exception { + final String expr = "{ (u, l) -> {" + + " if (l.shortName.lastIndexOf('.') > 0) {" + + " return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.'));" + + " }" + + " return false;" + + "} }"; + final BiFunction fn = generator.compile( + "lower-short-name-remove-ns", expr); + + assertNotNull(fn); + + final Service upper = new Service(); + upper.setShortName("svc-a"); + final Service lower = new Service(); + lower.setShortName("svc-a.ns1"); + assertTrue(fn.apply(upper, lower)); + + lower.setShortName("svc-b.ns1"); + assertFalse(fn.apply(upper, lower)); + + lower.setShortName("no-dot"); + assertFalse(fn.apply(upper, lower)); + } + + @Test + void compileLowerShortNameWithFqdn() throws Exception { + final String expr = "{ (u, l) -> {" + + " if (u.shortName.lastIndexOf(':') > 0) {" + + " return u.shortName.substring(0, u.shortName.lastIndexOf(':'))" + + " == l.shortName.concat('.svc.cluster.local');" + + " }" + + " return false;" + + "} }"; + final BiFunction fn = generator.compile( + "lower-short-name-with-fqdn", expr); + + assertNotNull(fn); + + final Service upper = new Service(); + upper.setShortName("svc-a.svc.cluster.local:8080"); + final Service lower = new Service(); + lower.setShortName("svc-a"); + assertTrue(fn.apply(upper, lower)); + + upper.setShortName("no-port"); + assertFalse(fn.apply(upper, lower)); + } +} diff --git a/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParserTest.java b/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParserTest.java new file mode 100644 index 000000000000..1cce3c327188 --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleScriptParserTest.java @@ -0,0 +1,159 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class HierarchyRuleScriptParserTest { + + @Test + void parseSimpleNameEquality() { + final HierarchyRuleModel model = HierarchyRuleScriptParser.parse( + "{ (u, l) -> u.name == l.name }"); + + assertEquals("u", model.getUpperParam()); + assertEquals("l", model.getLowerParam()); + assertInstanceOf(HierarchyRuleModel.SimpleComparison.class, model.getBody()); + + final HierarchyRuleModel.SimpleComparison cmp = + (HierarchyRuleModel.SimpleComparison) model.getBody(); + assertEquals(HierarchyRuleModel.CompareOp.EQ, cmp.getOp()); + + final HierarchyRuleModel.MethodChainExpr left = + (HierarchyRuleModel.MethodChainExpr) cmp.getLeft(); + assertEquals("u", left.getTarget()); + assertEquals(1, left.getSegments().size()); + assertEquals("name", left.getSegments().get(0).getName()); + + final HierarchyRuleModel.MethodChainExpr right = + (HierarchyRuleModel.MethodChainExpr) cmp.getRight(); + assertEquals("l", right.getTarget()); + assertEquals("name", right.getSegments().get(0).getName()); + } + + @Test + void parseShortNameEquality() { + final HierarchyRuleModel model = HierarchyRuleScriptParser.parse( + "{ (u, l) -> u.shortName == l.shortName }"); + + final HierarchyRuleModel.SimpleComparison cmp = + (HierarchyRuleModel.SimpleComparison) model.getBody(); + final HierarchyRuleModel.MethodChainExpr left = + (HierarchyRuleModel.MethodChainExpr) cmp.getLeft(); + assertEquals("shortName", left.getSegments().get(0).getName()); + } + + @Test + void parseLowerShortNameRemoveNs() { + // lower-short-name-remove-ns rule + final String expr = + "{ (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) " + + "return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); " + + "return false; } }"; + + final HierarchyRuleModel model = HierarchyRuleScriptParser.parse(expr); + assertEquals("u", model.getUpperParam()); + assertEquals("l", model.getLowerParam()); + assertInstanceOf(HierarchyRuleModel.BlockBody.class, model.getBody()); + + final HierarchyRuleModel.BlockBody block = + (HierarchyRuleModel.BlockBody) model.getBody(); + assertEquals(2, block.getStatements().size()); + + // First statement: if + final HierarchyRuleModel.IfStatement ifStmt = + (HierarchyRuleModel.IfStatement) block.getStatements().get(0); + assertInstanceOf( + HierarchyRuleModel.ComparisonCondition.class, ifStmt.getCondition()); + final HierarchyRuleModel.ComparisonCondition cond = + (HierarchyRuleModel.ComparisonCondition) ifStmt.getCondition(); + assertEquals(HierarchyRuleModel.CompareOp.GT, cond.getOp()); + + // Condition left: l.shortName.lastIndexOf('.') + final HierarchyRuleModel.MethodChainExpr condLeft = + (HierarchyRuleModel.MethodChainExpr) cond.getLeft(); + assertEquals("l", condLeft.getTarget()); + assertEquals(2, condLeft.getSegments().size()); + assertEquals("shortName", condLeft.getSegments().get(0).getName()); + assertInstanceOf( + HierarchyRuleModel.MethodCallSegment.class, condLeft.getSegments().get(1)); + final HierarchyRuleModel.MethodCallSegment lastIndexOf = + (HierarchyRuleModel.MethodCallSegment) condLeft.getSegments().get(1); + assertEquals("lastIndexOf", lastIndexOf.getName()); + assertEquals(1, lastIndexOf.getArguments().size()); + assertInstanceOf( + HierarchyRuleModel.StringLiteralExpr.class, lastIndexOf.getArguments().get(0)); + assertEquals(".", + ((HierarchyRuleModel.StringLiteralExpr) lastIndexOf.getArguments().get(0)).getValue()); + + // Then branch: return u.shortName == l.shortName.substring(0, ...) + assertEquals(1, ifStmt.getThenBranch().size()); + assertInstanceOf( + HierarchyRuleModel.ReturnStatement.class, ifStmt.getThenBranch().get(0)); + + // Second statement: return false + final HierarchyRuleModel.ReturnStatement retFalse = + (HierarchyRuleModel.ReturnStatement) block.getStatements().get(1); + assertInstanceOf(HierarchyRuleModel.BoolLiteralExpr.class, retFalse.getValue()); + final HierarchyRuleModel.BoolLiteralExpr falseExpr = + (HierarchyRuleModel.BoolLiteralExpr) retFalse.getValue(); + assertTrue(!falseExpr.isValue()); + } + + @Test + void parseLowerShortNameWithFqdn() { + // lower-short-name-with-fqdn rule + final String expr = + "{ (u, l) -> { if(u.shortName.lastIndexOf(':') > 0) " + + "return u.shortName.substring(0, u.shortName.lastIndexOf(':')) " + + "== l.shortName.concat('.svc.cluster.local'); " + + "return false; } }"; + + final HierarchyRuleModel model = HierarchyRuleScriptParser.parse(expr); + assertInstanceOf(HierarchyRuleModel.BlockBody.class, model.getBody()); + + final HierarchyRuleModel.BlockBody block = + (HierarchyRuleModel.BlockBody) model.getBody(); + assertEquals(2, block.getStatements().size()); + + // Verify the if condition checks u.shortName.lastIndexOf(':') > 0 + final HierarchyRuleModel.IfStatement ifStmt = + (HierarchyRuleModel.IfStatement) block.getStatements().get(0); + final HierarchyRuleModel.ComparisonCondition cond = + (HierarchyRuleModel.ComparisonCondition) ifStmt.getCondition(); + assertEquals(HierarchyRuleModel.CompareOp.GT, cond.getOp()); + + // Then branch has a return statement with == comparison + final HierarchyRuleModel.ReturnStatement retStmt = + (HierarchyRuleModel.ReturnStatement) ifStmt.getThenBranch().get(0); + // The return value should be a comparison (u.shortName.substring(...) == l.shortName.concat(...)) + // But since our grammar wraps returns as expressions, check the structure + assertInstanceOf(HierarchyRuleModel.Expr.class, retStmt.getValue()); + } + + @Test + void parseSyntaxErrorThrows() { + assertThrows(IllegalArgumentException.class, + () -> HierarchyRuleScriptParser.parse("{ invalid }")); + } +} diff --git a/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java b/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java deleted file mode 100644 index 5df94e71db9d..000000000000 --- a/oap-server/analyzer/lal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspiler.java +++ /dev/null @@ -1,923 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.server.transpiler.lal; - -import java.io.File; -import java.io.IOException; -import java.io.StringWriter; -import java.nio.file.Files; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.stream.Collectors; -import javax.tools.JavaCompiler; -import javax.tools.JavaFileObject; -import javax.tools.StandardJavaFileManager; -import javax.tools.ToolProvider; -import lombok.extern.slf4j.Slf4j; -import org.apache.skywalking.oap.server.transpiler.mal.MalToJavaTranspiler; -import org.codehaus.groovy.ast.ModuleNode; -import org.codehaus.groovy.ast.expr.ArgumentListExpression; -import org.codehaus.groovy.ast.expr.BinaryExpression; -import org.codehaus.groovy.ast.expr.BooleanExpression; -import org.codehaus.groovy.ast.expr.CastExpression; -import org.codehaus.groovy.ast.expr.ClosureExpression; -import org.codehaus.groovy.ast.expr.ConstantExpression; -import org.codehaus.groovy.ast.expr.Expression; -import org.codehaus.groovy.ast.expr.GStringExpression; -import org.codehaus.groovy.ast.expr.MapEntryExpression; -import org.codehaus.groovy.ast.expr.MapExpression; -import org.codehaus.groovy.ast.expr.MethodCallExpression; -import org.codehaus.groovy.ast.expr.NotExpression; -import org.codehaus.groovy.ast.expr.PropertyExpression; -import org.codehaus.groovy.ast.expr.TupleExpression; -import org.codehaus.groovy.ast.expr.VariableExpression; -import org.codehaus.groovy.ast.stmt.BlockStatement; -import org.codehaus.groovy.ast.stmt.EmptyStatement; -import org.codehaus.groovy.ast.stmt.ExpressionStatement; -import org.codehaus.groovy.ast.stmt.IfStatement; -import org.codehaus.groovy.ast.stmt.Statement; -import org.codehaus.groovy.control.CompilationUnit; -import org.codehaus.groovy.control.CompilerConfiguration; -import org.codehaus.groovy.control.Phases; -import org.codehaus.groovy.syntax.Types; - -/** - * Transpiles LAL (Log Analysis Language) Groovy scripts to Java source code at build time. - * Parses DSL strings into Groovy AST (via CompilationUnit at CONVERSION phase), - * walks AST nodes, and produces Java classes implementing LalExpression. - */ -@Slf4j -public class LalToJavaTranspiler { - - public static final String GENERATED_PACKAGE = - "org.apache.skywalking.oap.server.core.source.oal.rt.lal"; - - private static final Set CONSUMER_METHODS = Set.of( - "text", "extractor", "sink", "slowSql", "sampledTrace", "metrics", "sampler" - ); - - private static final Set STATIC_TYPES = Set.of("ProcessRegistry"); - - // ---- Batch state ---- - - private final Map lalSources = new LinkedHashMap<>(); - private final Map hashToClass = new LinkedHashMap<>(); - - private int tempVarCounter; - - /** - * Transpile a LAL DSL script to a Java class implementing LalExpression. - * - * @param className simple class name (e.g. "LalExpr_3") - * @param dslText the Groovy DSL string (filter { ... }) - * @return Java source code - */ - public String transpile(final String className, final String dslText) { - tempVarCounter = 0; - final ModuleNode module = parseToAST(dslText); - final Statement body = extractBody(module); - - final StringBuilder sb = new StringBuilder(); - emitHeader(sb, className); - - final List stmts = flattenStatements(body); - for (Statement stmt : stmts) { - emitStatement(sb, stmt, "filterSpec", "binding", 2); - } - - emitFooter(sb); - return sb.toString(); - } - - public void register(final String className, final String hash, final String source) { - lalSources.put(className, source); - hashToClass.put(hash, GENERATED_PACKAGE + "." + className); - } - - // ---- AST Parsing ---- - - ModuleNode parseToAST(final String expression) { - final CompilerConfiguration cc = new CompilerConfiguration(); - final CompilationUnit cu = new CompilationUnit(cc); - cu.addSource("Script", expression); - cu.compile(Phases.CONVERSION); - final List modules = cu.getAST().getModules(); - if (modules.isEmpty()) { - throw new IllegalStateException("No AST modules produced"); - } - return modules.get(0); - } - - Statement extractBody(final ModuleNode module) { - final BlockStatement block = module.getStatementBlock(); - if (block != null && !block.getStatements().isEmpty()) { - return block; - } - throw new IllegalStateException("Empty AST body"); - } - - // ---- Code Generation ---- - - private void emitHeader(final StringBuilder sb, final String className) { - sb.append("package ").append(GENERATED_PACKAGE).append(";\n\n"); - sb.append("import java.util.Map;\n"); - sb.append("import org.apache.skywalking.oap.log.analyzer.dsl.Binding;\n"); - sb.append("import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression;\n"); - sb.append("import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec;\n\n"); - sb.append("@SuppressWarnings(\"unchecked\")\n"); - sb.append("public class ").append(className).append(" implements LalExpression {\n\n"); - emitHelpers(sb); - sb.append(" @Override\n"); - sb.append(" public void execute(FilterSpec filterSpec, Binding binding) {\n"); - } - - private void emitHelpers(final StringBuilder sb) { - sb.append(" private static Object getAt(Object obj, String key) {\n"); - sb.append(" if (obj == null) return null;\n"); - sb.append(" if (obj instanceof Binding.Parsed) return ((Binding.Parsed) obj).getAt(key);\n"); - sb.append(" if (obj instanceof Map) return ((Map) obj).get(key);\n"); - sb.append(" return null;\n"); - sb.append(" }\n\n"); - - sb.append(" private static long toLong(Object obj) {\n"); - sb.append(" if (obj instanceof Number) return ((Number) obj).longValue();\n"); - sb.append(" if (obj instanceof String) return Long.parseLong((String) obj);\n"); - sb.append(" return 0L;\n"); - sb.append(" }\n\n"); - - sb.append(" private static int toInt(Object obj) {\n"); - sb.append(" if (obj instanceof Number) return ((Number) obj).intValue();\n"); - sb.append(" if (obj instanceof String) return Integer.parseInt((String) obj);\n"); - sb.append(" return 0;\n"); - sb.append(" }\n\n"); - - sb.append(" private static boolean toBoolean(Object obj) {\n"); - sb.append(" if (obj instanceof Boolean) return (Boolean) obj;\n"); - sb.append(" if (obj instanceof String) return Boolean.parseBoolean((String) obj);\n"); - sb.append(" return obj != null;\n"); - sb.append(" }\n\n"); - - sb.append(" private static boolean isTruthy(Object obj) {\n"); - sb.append(" if (obj == null) return false;\n"); - sb.append(" if (obj instanceof Boolean) return (Boolean) obj;\n"); - sb.append(" if (obj instanceof String) return !((String) obj).isEmpty();\n"); - sb.append(" if (obj instanceof Number) return ((Number) obj).doubleValue() != 0;\n"); - sb.append(" return true;\n"); - sb.append(" }\n\n"); - - sb.append(" private static boolean isNonEmptyString(Object obj) {\n"); - sb.append(" if (obj == null) return false;\n"); - sb.append(" String s = obj.toString();\n"); - sb.append(" return s != null && !s.trim().isEmpty();\n"); - sb.append(" }\n\n"); - } - - private void emitFooter(final StringBuilder sb) { - sb.append(" }\n"); - sb.append("}\n"); - } - - private void emitStatement(final StringBuilder sb, final Statement stmt, - final String receiver, final String bindingVar, final int indent) { - if (stmt instanceof BlockStatement) { - for (Statement s : ((BlockStatement) stmt).getStatements()) { - emitStatement(sb, s, receiver, bindingVar, indent); - } - } else if (stmt instanceof ExpressionStatement) { - emitExpressionStatement(sb, ((ExpressionStatement) stmt).getExpression(), - receiver, bindingVar, indent); - } else if (stmt instanceof IfStatement) { - emitIfStatement(sb, (IfStatement) stmt, receiver, bindingVar, indent); - } else if (stmt instanceof EmptyStatement) { - // skip - } else { - throw new UnsupportedOperationException( - "Unsupported statement: " + stmt.getClass().getSimpleName()); - } - } - - private void emitExpressionStatement(final StringBuilder sb, final Expression expr, - final String receiver, final String bindingVar, - final int indent) { - if (expr instanceof MethodCallExpression) { - emitMethodCall(sb, (MethodCallExpression) expr, receiver, bindingVar, indent); - } else if (expr instanceof BinaryExpression) { - emitBinaryAsNamedArg(sb, (BinaryExpression) expr, receiver, bindingVar, indent); - } else { - throw new UnsupportedOperationException( - "Unsupported expression statement: " + expr.getClass().getSimpleName() - + " (" + expr.getText() + ")"); - } - } - - private void emitMethodCall(final StringBuilder sb, final MethodCallExpression mce, - final String receiver, final String bindingVar, final int indent) { - final String methodName = mce.getMethodAsString(); - final Expression objExpr = mce.getObjectExpression(); - final ArgumentListExpression args = toArgList(mce.getArguments()); - final List argExprs = args.getExpressions(); - - // Top-level filter { ... } -> unwrap and emit body - if ("filter".equals(methodName) && isThisOrImplicit(objExpr)) { - if (!argExprs.isEmpty() && argExprs.get(0) instanceof ClosureExpression) { - final ClosureExpression closure = (ClosureExpression) argExprs.get(0); - emitStatement(sb, closure.getCode(), receiver, bindingVar, indent); - return; - } - } - - // json {} -> filterSpec.json() - if ("json".equals(methodName) && isThisOrImplicit(objExpr)) { - indent(sb, indent); - sb.append(receiver).append(".json();\n"); - return; - } - - // text { regexp /pattern/ } -> filterSpec.text(tp -> tp.regexp("pattern")) - if ("text".equals(methodName) && isThisOrImplicit(objExpr) - && !argExprs.isEmpty() && argExprs.get(0) instanceof ClosureExpression) { - final ClosureExpression closure = (ClosureExpression) argExprs.get(0); - final String tpVar = "tp"; - indent(sb, indent); - sb.append(receiver).append(".text(").append(tpVar).append(" -> {\n"); - emitStatement(sb, closure.getCode(), tpVar, bindingVar, indent + 1); - indent(sb, indent); - sb.append("});\n"); - return; - } - - // regexp /pattern/ -> tp.regexp("pattern") - if ("regexp".equals(methodName) && isThisOrImplicit(objExpr)) { - indent(sb, indent); - sb.append(receiver).append(".regexp("); - if (!argExprs.isEmpty()) { - sb.append(visitValueExpression(argExprs.get(0), bindingVar)); - } - sb.append(");\n"); - return; - } - - // Consumer overload methods: extractor, sink, slowSql, sampledTrace, metrics, sampler - if (CONSUMER_METHODS.contains(methodName) && isThisOrImplicit(objExpr) - && !argExprs.isEmpty() && argExprs.get(0) instanceof ClosureExpression) { - final ClosureExpression closure = (ClosureExpression) argExprs.get(0); - final String lambdaVar = lambdaVarFor(methodName); - final String childReceiver = childReceiverFor(lambdaVar); - indent(sb, indent); - sb.append(receiver).append(".").append(methodName).append("(") - .append(lambdaVar).append(" -> {\n"); - emitStatement(sb, closure.getCode(), childReceiver, bindingVar, indent + 1); - indent(sb, indent); - sb.append("});\n"); - return; - } - - // rateLimit("${expr}") { rpm N } -> sp.rateLimit(idExpr, rls -> rls.rpm(N)) - if ("rateLimit".equals(methodName) && isThisOrImplicit(objExpr)) { - final Expression idArg = argExprs.get(0); - final ClosureExpression closure = argExprs.size() > 1 - && argExprs.get(1) instanceof ClosureExpression - ? (ClosureExpression) argExprs.get(1) : null; - - indent(sb, indent); - sb.append(receiver).append(".rateLimit(") - .append(visitValueExpression(idArg, bindingVar)); - if (closure != null) { - final String rlsVar = "rls"; - sb.append(", ").append(rlsVar).append(" -> {\n"); - emitStatement(sb, closure.getCode(), rlsVar, bindingVar, indent + 1); - indent(sb, indent); - sb.append("}"); - } - sb.append(");\n"); - return; - } - - // abort {} -> filterSpec.abort() - if ("abort".equals(methodName) && isThisOrImplicit(objExpr)) { - indent(sb, indent); - sb.append(receiver).append(".abort();\n"); - return; - } - - // enforcer {} or dropper {} -> sink.enforcer() / sink.dropper() - if (("enforcer".equals(methodName) || "dropper".equals(methodName)) - && isThisOrImplicit(objExpr)) { - indent(sb, indent); - sb.append(receiver).append(".").append(methodName).append("();\n"); - return; - } - - // tag("KEY") as standalone statement - if ("tag".equals(methodName) && isThisOrImplicit(objExpr) - && !argExprs.isEmpty() && argExprs.get(0) instanceof ConstantExpression) { - indent(sb, indent); - sb.append(receiver).append(".tag(\"") - .append(MalToJavaTranspiler.escapeJava(argExprs.get(0).getText())) - .append("\");\n"); - return; - } - - // Simple value-setting methods: service(val), layer(val), timestamp(val), etc. - if (isThisOrImplicit(objExpr)) { - indent(sb, indent); - sb.append(receiver).append(".").append(methodName).append("("); - for (int i = 0; i < argExprs.size(); i++) { - if (i > 0) { - sb.append(", "); - } - sb.append(visitValueExpression(argExprs.get(i), bindingVar)); - } - sb.append(");\n"); - return; - } - - // Static method: ProcessRegistry.generateVirtualLocalProcess(...) - if (objExpr instanceof VariableExpression - && STATIC_TYPES.contains(((VariableExpression) objExpr).getName())) { - indent(sb, indent); - sb.append("org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry.") - .append(methodName).append("("); - for (int i = 0; i < argExprs.size(); i++) { - if (i > 0) { - sb.append(", "); - } - sb.append(visitValueExpression(argExprs.get(i), bindingVar)); - } - sb.append(");\n"); - return; - } - - throw new UnsupportedOperationException( - "Unsupported method call: " + methodName + " on " + objExpr.getClass().getSimpleName() - + " (" + mce.getText() + ")"); - } - - private void emitBinaryAsNamedArg(final StringBuilder sb, final BinaryExpression expr, - final String receiver, final String bindingVar, - final int indent) { - throw new UnsupportedOperationException( - "Unsupported binary expression as statement: " + expr.getText()); - } - - // ---- If/Else ---- - - private void emitIfStatement(final StringBuilder sb, final IfStatement ifStmt, - final String receiver, final String bindingVar, final int indent) { - indent(sb, indent); - sb.append("if ("); - sb.append(visitCondition(ifStmt.getBooleanExpression(), bindingVar)); - sb.append(") {\n"); - - emitStatement(sb, ifStmt.getIfBlock(), receiver, bindingVar, indent + 1); - - final Statement elseBlock = ifStmt.getElseBlock(); - if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { - indent(sb, indent); - if (elseBlock instanceof IfStatement) { - sb.append("} else "); - emitIfStatementInline(sb, (IfStatement) elseBlock, receiver, bindingVar, indent); - return; - } else { - sb.append("} else {\n"); - emitStatement(sb, elseBlock, receiver, bindingVar, indent + 1); - indent(sb, indent); - sb.append("}\n"); - } - } else { - indent(sb, indent); - sb.append("}\n"); - } - } - - private void emitIfStatementInline(final StringBuilder sb, final IfStatement ifStmt, - final String receiver, final String bindingVar, - final int indent) { - sb.append("if ("); - sb.append(visitCondition(ifStmt.getBooleanExpression(), bindingVar)); - sb.append(") {\n"); - - emitStatement(sb, ifStmt.getIfBlock(), receiver, bindingVar, indent + 1); - - final Statement elseBlock = ifStmt.getElseBlock(); - if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { - indent(sb, indent); - if (elseBlock instanceof IfStatement) { - sb.append("} else "); - emitIfStatementInline(sb, (IfStatement) elseBlock, receiver, bindingVar, indent); - } else { - sb.append("} else {\n"); - emitStatement(sb, elseBlock, receiver, bindingVar, indent + 1); - indent(sb, indent); - sb.append("}\n"); - } - } else { - indent(sb, indent); - sb.append("}\n"); - } - } - - // ---- Condition Visiting ---- - - private String visitCondition(final BooleanExpression boolExpr, final String bindingVar) { - return visitConditionExpr(boolExpr.getExpression(), bindingVar); - } - - String visitConditionExpr(final Expression expr, final String bindingVar) { - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - final int op = bin.getOperation().getType(); - - if (op == Types.COMPARE_EQUAL) { - final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); - final String right = visitValueExpression(bin.getRightExpression(), bindingVar); - if (bin.getRightExpression() instanceof ConstantExpression - && ((ConstantExpression) bin.getRightExpression()).getValue() instanceof String) { - return "\"" + MalToJavaTranspiler.escapeJava( - (String) ((ConstantExpression) bin.getRightExpression()).getValue()) - + "\".equals(" + left + ")"; - } - return "java.util.Objects.equals(" + left + ", " + right + ")"; - } - - if (op == Types.COMPARE_NOT_EQUAL) { - final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); - final String right = visitValueExpression(bin.getRightExpression(), bindingVar); - if (bin.getRightExpression() instanceof ConstantExpression - && ((ConstantExpression) bin.getRightExpression()).getValue() instanceof String) { - return "!\"" + MalToJavaTranspiler.escapeJava( - (String) ((ConstantExpression) bin.getRightExpression()).getValue()) - + "\".equals(" + left + ")"; - } - return "!java.util.Objects.equals(" + left + ", " + right + ")"; - } - - if (op == Types.COMPARE_LESS_THAN) { - final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); - final String right = visitValueExpression(bin.getRightExpression(), bindingVar); - return "toInt(" + left + ") < " + right; - } - - if (op == Types.COMPARE_GREATER_THAN_EQUAL) { - final String left = visitValueExpression(bin.getLeftExpression(), bindingVar); - final String right = visitValueExpression(bin.getRightExpression(), bindingVar); - return "toInt(" + left + ") >= " + right; - } - - if (op == Types.LOGICAL_AND) { - return visitConditionExpr(bin.getLeftExpression(), bindingVar) - + " && " + visitConditionExpr(bin.getRightExpression(), bindingVar); - } - - if (op == Types.LOGICAL_OR) { - return visitConditionExpr(bin.getLeftExpression(), bindingVar) - + " || " + visitConditionExpr(bin.getRightExpression(), bindingVar); - } - - throw new UnsupportedOperationException( - "Unsupported condition operator: " + bin.getOperation().getText()); - } - - if (expr instanceof NotExpression) { - final String inner = visitConditionExpr(((NotExpression) expr).getExpression(), bindingVar); - return "!" + inner; - } - - if (expr instanceof BooleanExpression) { - return visitConditionExpr(((BooleanExpression) expr).getExpression(), bindingVar); - } - - if (expr instanceof MethodCallExpression) { - final MethodCallExpression mce = (MethodCallExpression) expr; - final String methodName = mce.getMethodAsString(); - if ("toString".equals(methodName) || "trim".equals(methodName)) { - final String obj = visitValueExpression(mce.getObjectExpression(), bindingVar); - return "isNonEmptyString(" + obj + ")"; - } - return visitValueExpression(expr, bindingVar); - } - - return "isTruthy(" + visitValueExpression(expr, bindingVar) + ")"; - } - - // ---- Value Expression Visiting ---- - - String visitValueExpression(final Expression expr, final String bindingVar) { - if (expr instanceof ConstantExpression) { - return visitConstant((ConstantExpression) expr); - } - - if (expr instanceof VariableExpression) { - return visitVariable((VariableExpression) expr, bindingVar); - } - - if (expr instanceof PropertyExpression) { - return visitProperty((PropertyExpression) expr, bindingVar); - } - - if (expr instanceof CastExpression) { - return visitCast((CastExpression) expr, bindingVar); - } - - if (expr instanceof MethodCallExpression) { - return visitMethodCallValue((MethodCallExpression) expr, bindingVar); - } - - if (expr instanceof GStringExpression) { - return visitGString((GStringExpression) expr, bindingVar); - } - - if (expr instanceof MapExpression) { - return visitMapExpression((MapExpression) expr, bindingVar); - } - - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - final int op = bin.getOperation().getType(); - if (op == Types.COMPARE_EQUAL || op == Types.COMPARE_NOT_EQUAL - || op == Types.LOGICAL_AND || op == Types.LOGICAL_OR - || op == Types.COMPARE_LESS_THAN || op == Types.COMPARE_GREATER_THAN_EQUAL) { - return visitConditionExpr(expr, bindingVar); - } - } - - if (expr instanceof NotExpression) { - return "!" + visitValueExpression(((NotExpression) expr).getExpression(), bindingVar); - } - - throw new UnsupportedOperationException( - "Unsupported value expression: " + expr.getClass().getSimpleName() - + " (" + expr.getText() + ")"); - } - - private String visitConstant(final ConstantExpression expr) { - final Object value = expr.getValue(); - if (value instanceof String) { - return "\"" + MalToJavaTranspiler.escapeJava((String) value) + "\""; - } - if (value instanceof Integer) { - return value.toString(); - } - if (value instanceof Long) { - return value + "L"; - } - if (value instanceof Boolean) { - return value.toString(); - } - if (value instanceof Double) { - return value + "d"; - } - if (value == null) { - return "null"; - } - return value.toString(); - } - - private String visitVariable(final VariableExpression expr, final String bindingVar) { - final String name = expr.getName(); - if ("parsed".equals(name)) { - return bindingVar + ".parsed()"; - } - if ("log".equals(name)) { - return bindingVar + ".log()"; - } - if ("this".equals(name)) { - return "filterSpec"; - } - if (STATIC_TYPES.contains(name)) { - return "org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry"; - } - return name; - } - - private String visitProperty(final PropertyExpression expr, final String bindingVar) { - final Expression objExpr = expr.getObjectExpression(); - final String propName = expr.getPropertyAsString(); - final boolean isSafe = expr.isSafe(); - - final String obj = visitValueExpression(objExpr, bindingVar); - - // log.service -> binding.log().getService() - if (objExpr instanceof VariableExpression - && "log".equals(((VariableExpression) objExpr).getName())) { - return visitLogProperty(propName, bindingVar); - } - - // For parsed access and nested map access, use getAt() - if (isSafe) { - return "(" + obj + " == null ? null : getAt(" + obj + ", \"" + propName + "\"))"; - } - - return "getAt(" + obj + ", \"" + propName + "\")"; - } - - private String visitLogProperty(final String propName, final String bindingVar) { - switch (propName) { - case "service": - return bindingVar + ".log().getService()"; - case "serviceInstance": - return bindingVar + ".log().getServiceInstance()"; - case "endpoint": - return bindingVar + ".log().getEndpoint()"; - case "timestamp": - return bindingVar + ".log().getTimestamp()"; - default: - return bindingVar + ".log().get" + capitalize(propName) + "()"; - } - } - - private String visitCast(final CastExpression expr, final String bindingVar) { - final String inner = visitValueExpression(expr.getExpression(), bindingVar); - final String typeName = expr.getType().getName(); - - switch (typeName) { - case "java.lang.String": - case "String": - return "String.valueOf(" + inner + ")"; - case "java.lang.Long": - case "Long": - case "long": - return "toLong(" + inner + ")"; - case "java.lang.Integer": - case "Integer": - case "int": - return "toInt(" + inner + ")"; - case "java.lang.Boolean": - case "Boolean": - case "boolean": - return "toBoolean(" + inner + ")"; - default: - return "((" + typeName + ") " + inner + ")"; - } - } - - private String visitMethodCallValue(final MethodCallExpression mce, final String bindingVar) { - final String methodName = mce.getMethodAsString(); - final Expression objExpr = mce.getObjectExpression(); - final ArgumentListExpression args = toArgList(mce.getArguments()); - final boolean isSafe = mce.isSafe(); - - // tag("KEY") on filterSpec -> filterSpec.tag("KEY") - if ("tag".equals(methodName) && isThisOrImplicit(objExpr)) { - final List argExprs = args.getExpressions(); - if (!argExprs.isEmpty() && argExprs.get(0) instanceof ConstantExpression) { - return "filterSpec.tag(\"" - + MalToJavaTranspiler.escapeJava(argExprs.get(0).getText()) + "\")"; - } - } - - // ProcessRegistry.generateVirtualLocalProcess(...) - if (objExpr instanceof VariableExpression - && STATIC_TYPES.contains(((VariableExpression) objExpr).getName())) { - final StringBuilder sb = new StringBuilder(); - sb.append("org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry."); - sb.append(methodName).append("("); - final List argExprs = args.getExpressions(); - for (int i = 0; i < argExprs.size(); i++) { - if (i > 0) { - sb.append(", "); - } - sb.append(visitValueExpression(argExprs.get(i), bindingVar)); - } - sb.append(")"); - return sb.toString(); - } - - // toString(), trim() on safe navigation chain - if ("toString".equals(methodName) || "trim".equals(methodName)) { - final String obj = visitValueExpression(objExpr, bindingVar); - if (isSafe) { - return "(" + obj + " == null ? null : " + obj + "." + methodName + "())"; - } - return obj + "." + methodName + "()"; - } - - // Generic method call - final String obj = visitValueExpression(objExpr, bindingVar); - final StringBuilder sb = new StringBuilder(); - if (isSafe) { - sb.append("(").append(obj).append(" == null ? null : "); - } - sb.append(obj).append(".").append(methodName).append("("); - final List argExprs = args.getExpressions(); - for (int i = 0; i < argExprs.size(); i++) { - if (i > 0) { - sb.append(", "); - } - sb.append(visitValueExpression(argExprs.get(i), bindingVar)); - } - sb.append(")"); - if (isSafe) { - sb.append(")"); - } - return sb.toString(); - } - - private String visitGString(final GStringExpression expr, final String bindingVar) { - final List strings = expr.getStrings(); - final List values = expr.getValues(); - - final StringBuilder sb = new StringBuilder(); - for (int i = 0; i < strings.size(); i++) { - final String text = strings.get(i).getText(); - if (!text.isEmpty()) { - if (sb.length() > 0) { - sb.append(" + "); - } - sb.append("\"").append(MalToJavaTranspiler.escapeJava(text)).append("\""); - } - if (i < values.size()) { - final String val = visitValueExpression(values.get(i), bindingVar); - if (sb.length() > 0) { - sb.append(" + "); - } - sb.append(val); - } - } - return sb.length() > 0 ? sb.toString() : "\"\""; - } - - private String visitMapExpression(final MapExpression expr, final String bindingVar) { - final List entries = expr.getMapEntryExpressions(); - if (entries.isEmpty()) { - return "java.util.Collections.emptyMap()"; - } - if (entries.size() == 1) { - final MapEntryExpression e = entries.get(0); - return "Map.of(" + visitValueExpression(e.getKeyExpression(), bindingVar) - + ", " + visitValueExpression(e.getValueExpression(), bindingVar) + ")"; - } - final StringBuilder sb = new StringBuilder("Map.of("); - for (int i = 0; i < entries.size(); i++) { - if (i > 0) { - sb.append(", "); - } - final MapEntryExpression e = entries.get(i); - sb.append(visitValueExpression(e.getKeyExpression(), bindingVar)); - sb.append(", "); - sb.append(visitValueExpression(e.getValueExpression(), bindingVar)); - } - sb.append(")"); - return sb.toString(); - } - - // ---- Helpers ---- - - private List flattenStatements(final Statement stmt) { - if (stmt instanceof BlockStatement) { - return ((BlockStatement) stmt).getStatements(); - } - return List.of(stmt); - } - - private boolean isThisOrImplicit(final Expression expr) { - if (expr instanceof VariableExpression) { - final String name = ((VariableExpression) expr).getName(); - return "this".equals(name); - } - return false; - } - - private String lambdaVarFor(final String methodName) { - switch (methodName) { - case "extractor": return "ext"; - case "sink": return "s"; - case "slowSql": return "sql"; - case "sampledTrace": return "st"; - case "metrics": return "m"; - case "sampler": return "sp"; - case "text": return "tp"; - default: return "x"; - } - } - - private String childReceiverFor(final String lambdaVar) { - return lambdaVar; - } - - private ArgumentListExpression toArgList(final Expression args) { - if (args instanceof ArgumentListExpression) { - return (ArgumentListExpression) args; - } - if (args instanceof TupleExpression) { - final ArgumentListExpression ale = new ArgumentListExpression(); - for (Expression e : ((TupleExpression) args).getExpressions()) { - ale.addExpression(e); - } - return ale; - } - final ArgumentListExpression ale = new ArgumentListExpression(); - ale.addExpression(args); - return ale; - } - - private static String capitalize(final String s) { - if (s == null || s.isEmpty()) { - return s; - } - return Character.toUpperCase(s.charAt(0)) + s.substring(1); - } - - private static void indent(final StringBuilder sb, final int level) { - for (int i = 0; i < level; i++) { - sb.append(" "); - } - } - - // ---- Compilation & Manifest ---- - - /** - * Compile all registered LAL sources using javax.tools.JavaCompiler. - */ - public void compileAll(final File sourceDir, final File outputDir, - final String classpath) throws IOException { - if (lalSources.isEmpty()) { - log.info("No LAL sources to compile."); - return; - } - - final String packageDir = GENERATED_PACKAGE.replace('.', File.separatorChar); - final File srcPkgDir = new File(sourceDir, packageDir); - if (!srcPkgDir.exists() && !srcPkgDir.mkdirs()) { - throw new IOException("Failed to create source dir: " + srcPkgDir); - } - if (!outputDir.exists() && !outputDir.mkdirs()) { - throw new IOException("Failed to create output dir: " + outputDir); - } - - final List javaFiles = new ArrayList<>(); - for (Map.Entry entry : lalSources.entrySet()) { - final File javaFile = new File(srcPkgDir, entry.getKey() + ".java"); - Files.writeString(javaFile.toPath(), entry.getValue()); - javaFiles.add(javaFile); - } - - final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); - if (compiler == null) { - throw new IllegalStateException("No Java compiler available — requires JDK"); - } - - final StringWriter errorWriter = new StringWriter(); - - try (StandardJavaFileManager fileManager = compiler.getStandardFileManager(null, null, null)) { - final Iterable compilationUnits = - fileManager.getJavaFileObjectsFromFiles(javaFiles); - - final List options = Arrays.asList( - "-d", outputDir.getAbsolutePath(), - "-classpath", classpath - ); - - final JavaCompiler.CompilationTask task = compiler.getTask( - errorWriter, fileManager, null, options, null, compilationUnits); - - final boolean success = task.call(); - if (!success) { - for (Map.Entry entry : lalSources.entrySet()) { - log.error("Generated source for {}:\n{}", entry.getKey(), entry.getValue()); - } - throw new RuntimeException( - "Java compilation failed for " + javaFiles.size() + " LAL sources:\n" - + errorWriter); - } - } - - log.info("Compiled {} LAL sources to {}", lalSources.size(), outputDir); - } - - /** - * Write lal-expressions.txt manifest: hash=FQCN format. - */ - public void writeManifest(final File outputDir) throws IOException { - final File manifestDir = new File(outputDir, "META-INF"); - if (!manifestDir.exists() && !manifestDir.mkdirs()) { - throw new IOException("Failed to create META-INF dir: " + manifestDir); - } - - final List lines = hashToClass.entrySet().stream() - .map(e -> e.getKey() + "=" + e.getValue()) - .sorted() - .collect(Collectors.toList()); - Files.write(new File(manifestDir, "lal-expressions.txt").toPath(), lines); - log.info("Wrote lal-expressions.txt with {} entries", lines.size()); - } -} diff --git a/oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java b/oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java deleted file mode 100644 index 4a580a4bcd36..000000000000 --- a/oap-server/analyzer/lal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/lal/LalToJavaTranspilerTest.java +++ /dev/null @@ -1,678 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.server.transpiler.lal; - -import java.io.File; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.List; - -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; - -import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertTrue; - -class LalToJavaTranspilerTest { - - private LalToJavaTranspiler transpiler; - - @BeforeEach - void setUp() { - transpiler = new LalToJavaTranspiler(); - } - - // ---- Class Structure ---- - - @Test - void classStructure() { - final String java = transpiler.transpile("LalExpr_test", - "filter { json {} }"); - assertNotNull(java); - - assertTrue(java.contains("package " + LalToJavaTranspiler.GENERATED_PACKAGE), - "Should have correct package"); - assertTrue(java.contains("public class LalExpr_test implements LalExpression"), - "Should implement LalExpression"); - assertTrue(java.contains("public void execute(FilterSpec filterSpec, Binding binding)"), - "Should have execute method"); - } - - @Test - void helperMethodsPresent() { - final String java = transpiler.transpile("LalExpr_test", - "filter { json {} }"); - assertNotNull(java); - - assertTrue(java.contains("private static Object getAt(Object obj, String key)"), - "Should have getAt helper"); - assertTrue(java.contains("private static long toLong(Object obj)"), - "Should have toLong helper"); - assertTrue(java.contains("private static int toInt(Object obj)"), - "Should have toInt helper"); - assertTrue(java.contains("private static boolean toBoolean(Object obj)"), - "Should have toBoolean helper"); - assertTrue(java.contains("private static boolean isTruthy(Object obj)"), - "Should have isTruthy helper"); - assertTrue(java.contains("private static boolean isNonEmptyString(Object obj)"), - "Should have isNonEmptyString helper"); - } - - @Test - void importsPresent() { - final String java = transpiler.transpile("LalExpr_test", - "filter { json {} }"); - assertNotNull(java); - - assertTrue(java.contains("import org.apache.skywalking.oap.log.analyzer.dsl.Binding;"), - "Should import Binding"); - assertTrue(java.contains("import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression;"), - "Should import LalExpression"); - assertTrue(java.contains("import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec;"), - "Should import FilterSpec"); - } - - // ---- filter {} Unwrapping ---- - - @Test - void filterUnwrap() { - final String java = transpiler.transpile("LalExpr_test", - "filter { json {} }"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.json()"), - "Should unwrap filter block and call json() on filterSpec"); - assertTrue(!java.contains("filterSpec.filter("), - "Should NOT emit filterSpec.filter() call"); - } - - // ---- json {} ---- - - @Test - void jsonEmptyBlock() { - final String java = transpiler.transpile("LalExpr_test", - "filter { json {} }"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.json();"), - "Should emit no-arg json() call"); - } - - // ---- text { regexp $/pattern/$ } ---- - - @Test - void textWithRegexp() { - final String java = transpiler.transpile("LalExpr_test", - "filter { text { regexp $/(?\\d+)\\s+(?.+)/$ } }"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.text(tp -> {"), - "Should emit text with Consumer lambda"); - assertTrue(java.contains("tp.regexp("), - "Should call regexp on text parser spec"); - } - - // ---- extractor { ... } ---- - - @Test - void extractorWithService() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " service parsed.service as String\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.extractor(ext -> {"), - "Should emit extractor with Consumer lambda"); - assertTrue(java.contains("ext.service("), - "Should call service on extractor spec"); - assertTrue(java.contains("String.valueOf("), - "Should use String.valueOf for 'as String' cast"); - assertTrue(java.contains("getAt(binding.parsed(), \"service\")"), - "Should use getAt for parsed.service"); - } - - @Test - void extractorWithMultipleFields() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " service parsed.service as String\n" + - " instance parsed.instance as String\n" + - " endpoint parsed.endpoint as String\n" + - " layer parsed.layer as String\n" + - " timestamp parsed.time as String\n" + - " traceId parsed.traceId as String\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("ext.service(String.valueOf(getAt(binding.parsed(), \"service\")))"), - "Should extract service"); - assertTrue(java.contains("ext.instance(String.valueOf(getAt(binding.parsed(), \"instance\")))"), - "Should extract instance"); - assertTrue(java.contains("ext.endpoint(String.valueOf(getAt(binding.parsed(), \"endpoint\")))"), - "Should extract endpoint"); - assertTrue(java.contains("ext.layer(String.valueOf(getAt(binding.parsed(), \"layer\")))"), - "Should extract layer"); - assertTrue(java.contains("ext.timestamp(String.valueOf(getAt(binding.parsed(), \"time\")))"), - "Should extract timestamp"); - assertTrue(java.contains("ext.traceId(String.valueOf(getAt(binding.parsed(), \"traceId\")))"), - "Should extract traceId"); - } - - // ---- sink { ... } ---- - - @Test - void sinkWithEnforcer() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " sink {\n" + - " enforcer {}\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.sink(s -> {"), - "Should emit sink with Consumer lambda"); - assertTrue(java.contains("s.enforcer();"), - "Should emit no-arg enforcer() on sink spec"); - } - - @Test - void sinkWithDropper() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " sink {\n" + - " dropper {}\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("s.dropper();"), - "Should emit no-arg dropper() on sink spec"); - } - - @Test - void sinkWithSampler() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " sink {\n" + - " sampler {\n" + - " rateLimit('abc')\n" + - " }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("s.sampler(sp -> {"), - "Should emit sampler with Consumer lambda"); - assertTrue(java.contains("sp.rateLimit(\"abc\")"), - "Should call rateLimit on sampler spec"); - } - - // ---- abort {} ---- - - @Test - void abortBlock() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " abort {}\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.abort();"), - "Should emit no-arg abort() call"); - } - - // ---- parsed access ---- - - @Test - void parsedPropertyAccess() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { service parsed.service as String }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("binding.parsed()"), - "Should translate 'parsed' to binding.parsed()"); - assertTrue(java.contains("getAt(binding.parsed(), \"service\")"), - "Should use getAt for property access"); - } - - @Test - void parsedNestedAccess() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { service parsed.data.serviceName as String }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("getAt(getAt(binding.parsed(), \"data\"), \"serviceName\")"), - "Should translate nested parsed access to nested getAt()"); - } - - // ---- Safe navigation (?.) ---- - - @Test - void safeNavigationOnParsed() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { service parsed?.service as String }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("== null ? null : getAt("), - "Should translate ?. to null-safe ternary with getAt"); - } - - // ---- as Cast ---- - - @Test - void castAsString() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { service parsed.service as String }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("String.valueOf("), - "Should translate 'as String' to String.valueOf()"); - } - - @Test - void castAsLong() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { timestamp parsed.time as Long }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("toLong("), - "Should translate 'as Long' to toLong()"); - } - - // ---- log access ---- - - @Test - void logPropertyAccess() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { service log.service }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("binding.log().getService()"), - "Should translate log.service to binding.log().getService()"); - } - - // ---- if/else ---- - - @Test - void ifStatement() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.level == 'ERROR') {\n" + - " abort {}\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("if (\"ERROR\".equals(getAt(binding.parsed(), \"level\"))"), - "Should translate == with constant on left for null-safety"); - assertTrue(java.contains("filterSpec.abort()"), - "Should emit abort in if body"); - } - - @Test - void ifElseStatement() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.type == 'access') {\n" + - " extractor { layer 'HTTP' }\n" + - " } else {\n" + - " extractor { layer 'GENERAL' }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("if (\"access\".equals(getAt(binding.parsed(), \"type\"))"), - "Should have if condition"); - assertTrue(java.contains("} else {"), - "Should have else block"); - } - - @Test - void ifElseIfStatement() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.type == 'a') {\n" + - " extractor { layer 'A' }\n" + - " } else if (parsed.type == 'b') {\n" + - " extractor { layer 'B' }\n" + - " } else {\n" + - " extractor { layer 'C' }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("} else if ("), - "Should produce else-if chain"); - } - - // ---- Condition operators ---- - - @Test - void notEqualCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.status != 'ok') { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("!\"ok\".equals(getAt(binding.parsed(), \"status\"))"), - "Should translate != with negated .equals()"); - } - - @Test - void logicalAndCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.a == 'x' && parsed.b == 'y') { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("\"x\".equals(getAt(binding.parsed(), \"a\")) && \"y\".equals(getAt(binding.parsed(), \"b\"))"), - "Should translate && with .equals() on both sides"); - } - - @Test - void logicalOrCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.a == 'x' || parsed.a == 'y') { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("\"x\".equals(") && java.contains("|| \"y\".equals("), - "Should translate || correctly"); - } - - @Test - void truthinessCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.value) { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("isTruthy(getAt(binding.parsed(), \"value\"))"), - "Should translate bare expression to isTruthy()"); - } - - @Test - void negationCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (!parsed.value) { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("!isTruthy(getAt(binding.parsed(), \"value\"))"), - "Should translate !expr to negated isTruthy()"); - } - - @Test - void lessThanCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.code < 400) { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("toInt(getAt(binding.parsed(), \"code\")) < 400"), - "Should translate < with toInt on left"); - } - - @Test - void greaterThanEqualCondition() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (parsed.code >= 500) { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("toInt(getAt(binding.parsed(), \"code\")) >= 500"), - "Should translate >= with toInt on left"); - } - - // ---- GString interpolation ---- - - @Test - void gstringInterpolation() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor { service \"svc::${parsed.name}\" as String }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("\"svc::\""), - "Should have string prefix"); - assertTrue(java.contains("getAt(binding.parsed(), \"name\")"), - "Should have parsed access in interpolation"); - } - - // ---- tag(Map) ---- - - @Test - void tagWithMapLiteral() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " tag(status: parsed.status as String)\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("ext.tag("), - "Should call tag on extractor spec"); - } - - // ---- slowSql / sampledTrace / metrics ---- - - @Test - void slowSqlBlock() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " layer 'MYSQL'\n" + - " service parsed.service as String\n" + - " slowSql {\n" + - " id parsed.id as String\n" + - " statement parsed.statement as String\n" + - " latency parsed.latency as Long\n" + - " }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("ext.slowSql(sql -> {"), - "Should emit slowSql with Consumer lambda, var 'sql'"); - assertTrue(java.contains("sql.id("), - "Should call id on slowSql spec"); - assertTrue(java.contains("sql.statement("), - "Should call statement on slowSql spec"); - assertTrue(java.contains("sql.latency(toLong("), - "Should call latency with toLong"); - } - - @Test - void sampledTraceBlock() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " sampledTrace {\n" + - " uri parsed.uri as String\n" + - " latency parsed.latency as Long\n" + - " }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("ext.sampledTrace(st -> {"), - "Should emit sampledTrace with Consumer lambda, var 'st'"); - assertTrue(java.contains("st.uri("), - "Should call uri on sampledTrace spec"); - } - - @Test - void metricsBlock() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " metrics {\n" + - " name 'log_count'\n" + - " value 1\n" + - " }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("ext.metrics(m -> {"), - "Should emit metrics with Consumer lambda, var 'm'"); - assertTrue(java.contains("m.name(\"log_count\")"), - "Should call name on metrics spec"); - assertTrue(java.contains("m.value(1)"), - "Should call value on metrics spec"); - } - - // ---- Complete LAL Script ---- - - @Test - void completeScript() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " extractor {\n" + - " service parsed.service as String\n" + - " instance parsed.instance as String\n" + - " layer parsed.layer as String\n" + - " timestamp parsed.time as String\n" + - " }\n" + - " sink {\n" + - " enforcer {}\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.json();"), - "Should have json() call"); - assertTrue(java.contains("filterSpec.extractor(ext -> {"), - "Should have extractor block"); - assertTrue(java.contains("filterSpec.sink(s -> {"), - "Should have sink block"); - assertTrue(java.contains("s.enforcer();"), - "Should have enforcer in sink"); - } - - // ---- tag("KEY") as value ---- - - @Test - void tagAsValue() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " if (tag('status') == 'error') { abort {} }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("filterSpec.tag(\"status\")"), - "Should call tag on filterSpec"); - } - - // ---- rateLimit ---- - - @Test - void rateLimitWithClosureArg() { - final String java = transpiler.transpile("LalExpr_test", - "filter {\n" + - " json {}\n" + - " sink {\n" + - " sampler {\n" + - " rateLimit('myId') { rpm 5 }\n" + - " }\n" + - " }\n" + - "}"); - assertNotNull(java); - - assertTrue(java.contains("sp.rateLimit(\"myId\", rls -> {"), - "Should emit rateLimit with id and closure lambda"); - assertTrue(java.contains("rls.rpm(5)"), - "Should call rpm on rate limit spec"); - } - - // ---- Manifest ---- - - @Test - void manifest(@TempDir Path tempDir) throws Exception { - final String source = transpiler.transpile("LalExpr_a", "filter { json {} }"); - transpiler.register("LalExpr_a", "abc123hash", source); - - final File outputDir = tempDir.toFile(); - transpiler.writeManifest(outputDir); - - final File manifest = new File(outputDir, "META-INF/lal-expressions.txt"); - assertTrue(manifest.exists(), "Manifest file should exist"); - - final List lines = Files.readAllLines(manifest.toPath()); - assertTrue(lines.stream().anyMatch(l -> l.contains("abc123hash") && - l.contains(LalToJavaTranspiler.GENERATED_PACKAGE + ".LalExpr_a")), - "Should contain hash=FQCN mapping"); - } -} diff --git a/oap-server/analyzer/log-analyzer-v2/pom.xml b/oap-server/analyzer/log-analyzer-v2/pom.xml deleted file mode 100644 index 6456f8209f16..000000000000 --- a/oap-server/analyzer/log-analyzer-v2/pom.xml +++ /dev/null @@ -1,72 +0,0 @@ - - - - - - analyzer - org.apache.skywalking - ${revision} - - 4.0.0 - - log-analyzer-v2 - Pure Java LAL runtime that loads transpiled LalExpression classes from manifest instead of Groovy - - - - org.apache.skywalking - log-analyzer - ${project.version} - - - - - - - org.apache.maven.plugins - maven-shade-plugin - - - package - - shade - - - true - - - org.apache.skywalking:log-analyzer - - - - - org.apache.skywalking:log-analyzer - - org/apache/skywalking/oap/log/analyzer/dsl/DSL.class - org/apache/skywalking/oap/log/analyzer/dsl/DSL$*.class - - - - - - - - - - diff --git a/oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java b/oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java deleted file mode 100644 index b3f4e4c977d2..000000000000 --- a/oap-server/analyzer/log-analyzer-v2/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java +++ /dev/null @@ -1,142 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.log.analyzer.dsl; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.nio.charset.StandardCharsets; -import java.security.MessageDigest; -import java.security.NoSuchAlgorithmException; -import java.util.HashMap; -import java.util.Map; -import java.util.concurrent.atomic.AtomicInteger; -import lombok.AccessLevel; -import lombok.RequiredArgsConstructor; -import lombok.extern.slf4j.Slf4j; -import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; -import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; -import org.apache.skywalking.oap.server.library.module.ModuleManager; -import org.apache.skywalking.oap.server.library.module.ModuleStartException; - -/** - * Same-FQCN replacement for upstream LAL DSL. - * Loads pre-compiled {@link LalExpression} classes from lal-expressions.txt manifest - * (keyed by SHA-256 hash) instead of Groovy {@code GroovyShell} runtime compilation. - */ -@Slf4j -@RequiredArgsConstructor(access = AccessLevel.PRIVATE) -public class DSL { - private static final String MANIFEST_PATH = "META-INF/lal-expressions.txt"; - private static volatile Map EXPRESSION_MAP; - private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); - - private final LalExpression expression; - private final FilterSpec filterSpec; - private Binding binding; - - public static DSL of(final ModuleManager moduleManager, - final LogAnalyzerModuleConfig config, - final String dsl) throws ModuleStartException { - final Map exprMap = loadManifest(); - final String dslHash = sha256(dsl); - final String className = exprMap.get(dslHash); - if (className == null) { - throw new ModuleStartException( - "Pre-compiled LAL expression not found for DSL hash: " + dslHash - + ". Available: " + exprMap.size() + " expressions."); - } - - try { - final Class exprClass = Class.forName(className); - final LalExpression expression = (LalExpression) exprClass.getDeclaredConstructor().newInstance(); - final FilterSpec filterSpec = new FilterSpec(moduleManager, config); - final int count = LOADED_COUNT.incrementAndGet(); - log.debug("Loaded pre-compiled LAL expression [{}/{}]: {}", count, exprMap.size(), className); - return new DSL(expression, filterSpec); - } catch (ClassNotFoundException e) { - throw new ModuleStartException( - "Pre-compiled LAL expression class not found: " + className, e); - } catch (ReflectiveOperationException e) { - throw new ModuleStartException( - "Pre-compiled LAL expression instantiation failed: " + className, e); - } - } - - public void bind(final Binding binding) { - this.binding = binding; - this.filterSpec.bind(binding); - } - - public void evaluate() { - expression.execute(filterSpec, binding); - } - - private static Map loadManifest() { - if (EXPRESSION_MAP != null) { - return EXPRESSION_MAP; - } - synchronized (DSL.class) { - if (EXPRESSION_MAP != null) { - return EXPRESSION_MAP; - } - final Map map = new HashMap<>(); - try (InputStream is = DSL.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { - if (is == null) { - log.warn("LAL expression manifest not found: {}", MANIFEST_PATH); - EXPRESSION_MAP = map; - return map; - } - try (BufferedReader reader = new BufferedReader( - new InputStreamReader(is, StandardCharsets.UTF_8))) { - String line; - while ((line = reader.readLine()) != null) { - line = line.trim(); - if (line.isEmpty()) { - continue; - } - final String[] parts = line.split("=", 2); - if (parts.length == 2) { - map.put(parts[0], parts[1]); - } - } - } - } catch (IOException e) { - throw new IllegalStateException("Failed to load LAL expression manifest", e); - } - log.info("Loaded {} pre-compiled LAL expressions from manifest", map.size()); - EXPRESSION_MAP = map; - return map; - } - } - - static String sha256(final String input) { - try { - final MessageDigest digest = MessageDigest.getInstance("SHA-256"); - final byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8)); - final StringBuilder hex = new StringBuilder(); - for (final byte b : hash) { - hex.append(String.format("%02x", b)); - } - return hex.toString(); - } catch (NoSuchAlgorithmException e) { - throw new IllegalStateException("SHA-256 not available", e); - } - } -} diff --git a/oap-server/analyzer/log-analyzer/CLAUDE.md b/oap-server/analyzer/log-analyzer/CLAUDE.md new file mode 100644 index 000000000000..a7b60c4a8e50 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/CLAUDE.md @@ -0,0 +1,128 @@ +# LAL Compiler + +Compiles LAL (Log Analysis Language) scripts into `LalExpression` implementation classes at runtime using ANTLR4 parsing and Javassist bytecode generation. + +## Compilation Workflow + +``` +LAL DSL string + → LALScriptParser.parse(dsl) [ANTLR4 lexer/parser → listener] + → LALScriptModel (immutable AST) + → LALClassGenerator.compileFromModel(model) + Phase 1: collectConsumers(model) — pre-scan for blocks needing Consumer callbacks + Phase 2: compileConsumerClass() — generate each consumer as separate Javassist class + Phase 3: classPool.makeClass() — create main class implementing LalExpression + Phase 4: generateExecuteMethod() — emit Java source referencing this._consumerN fields + Phase 5: ctClass.toClass(LalExpressionPackageHolder.class) + wire consumer fields + → LalExpression instance +``` + +The generated class implements: +```java +void execute(Object filterSpec, Object binding) + // cast internally to FilterSpec and Binding +``` + +## File Structure + +``` +oap-server/analyzer/log-analyzer/ + src/main/antlr4/.../LALLexer.g4 — ANTLR4 lexer grammar + src/main/antlr4/.../LALParser.g4 — ANTLR4 parser grammar + + src/main/java/.../compiler/ + LALScriptParser.java — ANTLR4 facade: DSL string → AST + LALScriptModel.java — Immutable AST model classes + LALClassGenerator.java — Javassist code generator + rt/ + LalExpressionPackageHolder.java — Class loading anchor (empty marker) + BindingAware.java — Interface for consumers needing Binding access + + src/test/java/.../compiler/ + LALScriptParserTest.java — 8 parser tests + LALClassGeneratorTest.java — 6 generator tests +``` + +## Package & Class Naming + +| Component | Package / Name | +|-----------|---------------| +| Parser/Model/Generator | `org.apache.skywalking.oap.log.analyzer.compiler` | +| Generated classes | `org.apache.skywalking.oap.log.analyzer.compiler.rt.LalExpr_` | +| Consumer classes | `org.apache.skywalking.oap.log.analyzer.compiler.rt.LalExpr__C` | +| Package holder | `org.apache.skywalking.oap.log.analyzer.compiler.rt.LalExpressionPackageHolder` | +| Binding aware | `org.apache.skywalking.oap.log.analyzer.compiler.rt.BindingAware` | +| Functional interface | `org.apache.skywalking.oap.log.analyzer.dsl.LalExpression` (in log-analyzer) | + +`` is a global `AtomicInteger` counter. `` is the consumer index within the script. + +## Consumer Pattern (BindingAware) + +LAL's FilterSpec API uses `Consumer` callbacks: `filterSpec.extractor(Consumer)`, `filterSpec.sink(Consumer)`, etc. Since Javassist cannot compile anonymous inner classes, consumers are pre-compiled as separate classes. + +Each consumer class implements both `java.util.function.Consumer` and `BindingAware`: +- `BindingAware.setBinding(Binding)` — called before each FilterSpec method to inject the current Binding +- `Consumer.accept(Object)` — casts to the specific Spec type and executes the block body + +The main class's `execute()` method emits: +```java +((BindingAware) this._consumer0).setBinding(binding); +filterSpec.extractor(this._consumer0); +``` + +Consumer traversal order in `collectConsumers()` must exactly match the order in `generateFilterStatement()`. + +## Javassist Constraints + +- **No anonymous inner classes**: All `Consumer` callbacks pre-compiled as separate `CtClass` instances. +- **No lambda expressions**: Same workaround as above. +- **Spec class packages**: Parser specs are in `dsl.spec.parser.*` (not `dsl.spec.extractor.*`): + - `spec.parser.TextParserSpec`, `spec.parser.JsonParserSpec`, `spec.parser.YamlParserSpec` + - `spec.extractor.ExtractorSpec` + - `spec.extractor.slowsql.SlowSqlSpec`, `spec.extractor.sampledtrace.SampledTraceSpec` + - `spec.sink.SinkSpec`, `spec.sink.SamplerSpec` + +## Example + +**Input**: `filter { json {} extractor { service parsed.service as String } sink {} }` + +Three classes are generated: + +1. **`LalExpr_0_C0`** — Consumer for extractor block: + ```java + // implements Consumer, BindingAware + public void accept(Object arg) { + ExtractorSpec _t = (ExtractorSpec) arg; + _t.service(String.valueOf(getAt(binding.parsed(), "service"))); + } + ``` + +2. **`LalExpr_0`** — Main class implementing `LalExpression`: + ```java + public Consumer _consumer0; // wired after toClass() + + public void execute(Object arg0, Object arg1) { + FilterSpec filterSpec = (FilterSpec) arg0; + Binding binding = (Binding) arg1; + filterSpec.json(); + ((BindingAware) this._consumer0).setBinding(binding); + filterSpec.extractor(this._consumer0); + filterSpec.sink(); + } + ``` + +**Consumer allocation rules**: +- `json {}` with no `abortOnFailure` → no consumer, emits `filterSpec.json()` +- `json { abortOnFailure }` → allocates a consumer +- `text { regexp '...' }` → allocates a consumer +- `text {}` with no regexp → no consumer, emits `filterSpec.text()` +- `extractor { ... }` → always allocates a consumer +- `sink {}` empty → no consumer, emits `filterSpec.sink()` +- `sink { enforcer {} }` → allocates a consumer + +## Dependencies + +All within this module (grammar, compiler, and runtime are merged): +- ANTLR4 grammar → generates lexer/parser at build time +- `LalExpression`, `Binding`, `FilterSpec`, all Spec classes — in `dsl` package of this module +- `javassist` — bytecode generation diff --git a/oap-server/analyzer/log-analyzer/pom.xml b/oap-server/analyzer/log-analyzer/pom.xml index 576539cb8594..ff811b15960b 100644 --- a/oap-server/analyzer/log-analyzer/pom.xml +++ b/oap-server/analyzer/log-analyzer/pom.xml @@ -7,13 +7,14 @@ ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at ~ - ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. + ~ --> @@ -25,7 +26,6 @@ 4.0.0 log-analyzer - jar @@ -43,13 +43,34 @@ agent-analyzer ${project.version} - - org.apache.groovy - groovy - com.fasterxml.jackson.core jackson-databind + + org.antlr + antlr4-runtime + + + org.javassist + javassist + + + + + + org.antlr + antlr4-maven-plugin + + + antlr + + antlr4 + + + + + + diff --git a/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALLexer.g4 b/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALLexer.g4 new file mode 100644 index 000000000000..4df6f7113ab1 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALLexer.g4 @@ -0,0 +1,175 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +// Log Analysis Language lexer +lexer grammar LALLexer; + +@Header {package org.apache.skywalking.lal.rt.grammar;} + +// Keywords - block structure +FILTER: 'filter'; +TEXT: 'text'; +JSON: 'json'; +YAML: 'yaml'; +EXTRACTOR: 'extractor'; +SINK: 'sink'; +ABORT: 'abort'; + +// Keywords - extractor statements +SERVICE: 'service'; +INSTANCE: 'instance'; +ENDPOINT: 'endpoint'; +LAYER: 'layer'; +TRACE_ID: 'traceId'; +SEGMENT_ID: 'segmentId'; +SPAN_ID: 'spanId'; +TIMESTAMP: 'timestamp'; +TAG: 'tag'; +METRICS: 'metrics'; +SLOW_SQL: 'slowSql'; +SAMPLED_TRACE: 'sampledTrace'; +REGEXP: 'regexp'; +ABORT_ON_FAILURE: 'abortOnFailure'; +NAME: 'name'; +VALUE: 'value'; +LABELS: 'labels'; +ID: 'id'; +STATEMENT: 'statement'; +LATENCY: 'latency'; +URI: 'uri'; +REASON: 'reason'; +PROCESS_ID: 'processId'; +DEST_PROCESS_ID: 'destProcessId'; +DETECT_POINT: 'detectPoint'; +COMPONENT_ID: 'componentId'; +REPORT_SERVICE: 'reportService'; + +// Keywords - sink statements +SAMPLER: 'sampler'; +RATE_LIMIT: 'rateLimit'; +RPM: 'rpm'; +ENFORCER: 'enforcer'; +DROPPER: 'dropper'; + +// Keywords - control flow +IF: 'if'; +ELSE: 'else'; + +// Keywords - type cast +AS: 'as'; +STRING_TYPE: 'String'; +LONG_TYPE: 'Long'; +INTEGER_TYPE: 'Integer'; +BOOLEAN_TYPE: 'Boolean'; + +// Keywords - built-in references +LOG: 'log'; +PARSED: 'parsed'; + +// Keywords - utility class references +PROCESS_REGISTRY: 'ProcessRegistry'; + +// Comparison and logical operators +DEQ: '=='; +NEQ: '!='; +AND: '&&'; +OR: '||'; +NOT: '!'; +GT: '>'; +LT: '<'; +GTE: '>='; +LTE: '<='; + +// Delimiters +DOT: '.'; +COMMA: ','; +COLON: ':'; +SEMI: ';'; +L_PAREN: '('; +R_PAREN: ')'; +L_BRACE: '{'; +R_BRACE: '}'; +L_BRACKET: '['; +R_BRACKET: ']'; +QUESTION: '?'; +ASSIGN: '='; + +// Arithmetic +PLUS: '+'; +MINUS: '-'; +STAR: '*'; +SLASH: '/'; + +// Literals +TRUE: 'true'; +FALSE: 'false'; +NULL: 'null'; + +NUMBER + : Digit+ ('.' Digit+)? + ; + +// String literal: single or double quoted +STRING + : '\'' (~['\\\r\n] | EscapeSequence)* '\'' + | '"' (~["\\\r\n] | EscapeSequence)* '"' + ; + +// Groovy-style slashy string for regex patterns: $/pattern/$ +SLASHY_STRING + : '$/' .*? '/$' + ; + +// Comments +LINE_COMMENT + : '//' ~[\r\n]* -> channel(HIDDEN) + ; + +BLOCK_COMMENT + : '/*' .*? '*/' -> channel(HIDDEN) + ; + +// Whitespace +WS + : [ \t\r\n]+ -> channel(HIDDEN) + ; + +// Identifiers +IDENTIFIER + : Letter LetterOrDigit* + ; + +// Fragments +fragment EscapeSequence + : '\\' [btnfr"'\\] + | '\\' ([0-3]? [0-7])? [0-7] + | '\\' . // catch-all for regex escapes like \d, \w, \s + ; + +fragment Digit + : [0-9] + ; + +fragment Letter + : [a-zA-Z_] + ; + +fragment LetterOrDigit + : Letter + | [0-9] + ; diff --git a/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 b/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 new file mode 100644 index 000000000000..b381aa157369 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 @@ -0,0 +1,464 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +// Log Analysis Language parser +// +// Covers LAL DSL patterns: +// filter { parser {} extractor {} sink {} } +// if (tag("LOG_KIND") == "NGINX_ACCESS_LOG") { ... } +// text { regexp $/pattern/$ } +// json { abortOnFailure true } +// extractor { service parsed.service as String; tag 'key': value } +// metrics { name "metric_name"; value 1; labels key: val } +// slowSql { id parsed.id as String; statement parsed.statement as String; latency parsed.query_time as Long } +// sampledTrace { latency parsed.latency as Long; uri parsed.uri as String; ... } +// sink { sampler { rateLimit("id") { rpm 6000 } } } +parser grammar LALParser; + +@Header {package org.apache.skywalking.lal.rt.grammar;} + +options { tokenVocab=LALLexer; } + +// ==================== Top-level ==================== + +root + : filterBlock EOF + ; + +// ==================== Filter block ==================== + +filterBlock + : FILTER L_BRACE filterContent R_BRACE + ; + +filterContent + : filterStatement* + ; + +filterStatement + : parserBlock + | extractorBlock + | sinkBlock + | ifStatement + | abortBlock + ; + +// ==================== Parser blocks ==================== + +parserBlock + : textBlock + | jsonBlock + | yamlBlock + ; + +textBlock + : TEXT L_BRACE textContent R_BRACE + ; + +textContent + : (regexpStatement | abortOnFailureStatement)* + ; + +regexpStatement + : REGEXP regexpPattern + ; + +regexpPattern + : SLASHY_STRING + | STRING + ; + +jsonBlock + : JSON L_BRACE jsonContent R_BRACE + ; + +jsonContent + : abortOnFailureStatement? + ; + +yamlBlock + : YAML L_BRACE yamlContent R_BRACE + ; + +yamlContent + : abortOnFailureStatement? + ; + +abortOnFailureStatement + : ABORT_ON_FAILURE boolValue + ; + +abortBlock + : ABORT L_BRACE R_BRACE + ; + +// ==================== Extractor block ==================== + +extractorBlock + : EXTRACTOR L_BRACE extractorContent R_BRACE + ; + +extractorContent + : extractorStatement* + ; + +extractorStatement + : serviceStatement + | instanceStatement + | endpointStatement + | layerStatement + | traceIdStatement + | segmentIdStatement + | spanIdStatement + | timestampStatement + | tagStatement + | metricsBlock + | slowSqlBlock + | sampledTraceBlock + | ifStatement + ; + +serviceStatement + : SERVICE valueAccess typeCast? + ; + +instanceStatement + : INSTANCE valueAccess typeCast? + ; + +endpointStatement + : ENDPOINT valueAccess typeCast? + ; + +layerStatement + : LAYER valueAccess typeCast? + ; + +traceIdStatement + : TRACE_ID valueAccess typeCast? + ; + +segmentIdStatement + : SEGMENT_ID valueAccess typeCast? + ; + +spanIdStatement + : SPAN_ID valueAccess typeCast? + ; + +timestampStatement + : TIMESTAMP valueAccess typeCast? (COMMA STRING)? + ; + +tagStatement + : TAG tagMap + | TAG STRING COLON valueAccess typeCast? + ; + +tagMap + : anyIdentifier COLON valueAccess typeCast? (COMMA anyIdentifier COLON valueAccess typeCast?)* + ; + +// ==================== Metrics block ==================== + +metricsBlock + : METRICS L_BRACE metricsContent R_BRACE + ; + +metricsContent + : metricsStatement* + ; + +metricsStatement + : metricsNameStatement + | metricsTimestampStatement + | metricsLabelsStatement + | metricsValueStatement + ; + +metricsNameStatement + : NAME valueAccess typeCast? + ; + +metricsTimestampStatement + : TIMESTAMP valueAccess typeCast? + ; + +metricsLabelsStatement + : LABELS labelMap + ; + +labelMap + : labelEntry (COMMA labelEntry)* + ; + +labelEntry + : anyIdentifier COLON valueAccess typeCast? + ; + +metricsValueStatement + : VALUE valueAccess typeCast? + ; + +// ==================== Slow SQL block ==================== + +slowSqlBlock + : SLOW_SQL L_BRACE slowSqlContent R_BRACE + ; + +slowSqlContent + : slowSqlStatement* + ; + +slowSqlStatement + : slowSqlIdStatement + | slowSqlStatementStatement + | slowSqlLatencyStatement + ; + +slowSqlIdStatement + : ID valueAccess typeCast? + ; + +slowSqlStatementStatement + : STATEMENT valueAccess typeCast? + ; + +slowSqlLatencyStatement + : LATENCY valueAccess typeCast? + ; + +// ==================== Sampled trace block ==================== + +sampledTraceBlock + : SAMPLED_TRACE L_BRACE sampledTraceContent R_BRACE + ; + +sampledTraceContent + : sampledTraceStatement* + ; + +sampledTraceStatement + : sampledTraceLatencyStatement + | sampledTraceUriStatement + | sampledTraceReasonStatement + | sampledTraceProcessIdStatement + | sampledTraceDestProcessIdStatement + | sampledTraceDetectPointStatement + | sampledTraceComponentIdStatement + | reportServiceStatement + | ifStatement + ; + +sampledTraceLatencyStatement + : LATENCY valueAccess typeCast? + ; + +sampledTraceUriStatement + : URI valueAccess typeCast? + ; + +sampledTraceReasonStatement + : REASON valueAccess typeCast? + ; + +sampledTraceProcessIdStatement + : PROCESS_ID valueAccess typeCast? + ; + +sampledTraceDestProcessIdStatement + : DEST_PROCESS_ID valueAccess typeCast? + ; + +sampledTraceDetectPointStatement + : DETECT_POINT valueAccess typeCast? + ; + +sampledTraceComponentIdStatement + : COMPONENT_ID valueAccess typeCast? + ; + +reportServiceStatement + : REPORT_SERVICE valueAccess typeCast? + ; + +// ==================== Sink block ==================== + +sinkBlock + : SINK L_BRACE sinkContent R_BRACE + ; + +sinkContent + : sinkStatement* + ; + +sinkStatement + : samplerBlock + | enforcerStatement + | dropperStatement + | ifStatement + ; + +samplerBlock + : SAMPLER L_BRACE samplerContent R_BRACE + ; + +samplerContent + : (rateLimitBlock | ifStatement)* + ; + +rateLimitBlock + : RATE_LIMIT L_PAREN rateLimitId R_PAREN L_BRACE rateLimitContent R_BRACE + ; + +rateLimitId + : STRING + ; + +rateLimitContent + : RPM NUMBER + ; + +enforcerStatement + : ENFORCER L_BRACE R_BRACE + ; + +dropperStatement + : DROPPER L_BRACE R_BRACE + ; + +// ==================== Control flow ==================== + +ifStatement + : IF L_PAREN condition R_PAREN L_BRACE + ifBody + R_BRACE + (ELSE IF L_PAREN condition R_PAREN L_BRACE + ifBody + R_BRACE)* + (ELSE L_BRACE + ifBody + R_BRACE)? + ; + +ifBody + : filterStatement* + | extractorStatement* + | sinkStatement* + | sampledTraceStatement* + | samplerContent + ; + +// ==================== Conditions ==================== + +condition + : condition AND condition # condAnd + | condition OR condition # condOr + | NOT condition # condNot + | L_PAREN condition R_PAREN # condParen + | conditionExpr DEQ conditionExpr # condEq + | conditionExpr NEQ conditionExpr # condNeq + | conditionExpr GT conditionExpr # condGt + | conditionExpr LT conditionExpr # condLt + | conditionExpr GTE conditionExpr # condGte + | conditionExpr LTE conditionExpr # condLte + | conditionExpr # condSingle + ; + +conditionExpr + : valueAccess typeCast? # condValueAccess + | STRING # condString + | NUMBER # condNumber + | boolValue # condBool + | NULL # condNull + | functionInvocation # condFunctionCall + ; + +// ==================== Value access ==================== + +// Accessing parsed values, log fields, and method calls: +// parsed.level, parsed?.response?.responseCode?.value +// log.service, log.timestamp, log.serviceInstance +// tag("LOG_KIND") +// ProcessRegistry.generateVirtualLocalProcess(...) + +valueAccess + : valueAccessPrimary (valueAccessSegment)* + ; + +valueAccessPrimary + : PARSED # valueParsed + | LOG # valueLog + | IDENTIFIER # valueIdentifier + | STRING # valueString + | NUMBER # valueNumber + | boolValue # valueBool + | NULL # valueNull + | functionInvocation # valueFunctionCall + ; + +valueAccessSegment + : DOT anyIdentifier # segmentField + | QUESTION DOT anyIdentifier # segmentSafeField + | DOT functionInvocation # segmentMethod + | QUESTION DOT functionInvocation # segmentSafeMethod + ; + +functionInvocation + : IDENTIFIER L_PAREN functionArgList? R_PAREN + ; + +functionArgList + : functionArg (COMMA functionArg)* + ; + +functionArg + : valueAccess typeCast? + | STRING + | NUMBER + | boolValue + | NULL + ; + +// ==================== Type cast ==================== + +typeCast + : AS (STRING_TYPE | LONG_TYPE | INTEGER_TYPE | BOOLEAN_TYPE) + ; + +// ==================== Common ==================== + +// Allows keywords to be used as identifiers in contexts like field names, +// labels, and value access segments (e.g. parsed.service, parsed.layer). +anyIdentifier + : IDENTIFIER + | SERVICE | INSTANCE | ENDPOINT | LAYER + | TRACE_ID | SEGMENT_ID | SPAN_ID | TIMESTAMP + | TAG | METRICS | SLOW_SQL | SAMPLED_TRACE + | REGEXP | ABORT_ON_FAILURE + | NAME | VALUE | LABELS + | ID | STATEMENT | LATENCY + | URI | REASON | PROCESS_ID | DEST_PROCESS_ID + | DETECT_POINT | COMPONENT_ID | REPORT_SERVICE + | SAMPLER | RATE_LIMIT | RPM | ENFORCER | DROPPER + | TEXT | JSON | YAML | FILTER | EXTRACTOR | SINK | ABORT + ; + +boolValue + : TRUE | FALSE + ; diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java new file mode 100644 index 000000000000..092d9f45f126 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java @@ -0,0 +1,627 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; +import javassist.ClassPool; +import javassist.CtClass; +import javassist.CtField; +import javassist.CtNewConstructor; +import javassist.CtNewMethod; +import org.apache.skywalking.oap.log.analyzer.compiler.rt.LalExpressionPackageHolder; +import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression; + +/** + * Generates {@link LalExpression} implementation classes from + * {@link LALScriptModel} AST using Javassist bytecode generation. + * + *

Because Javassist cannot compile anonymous inner classes, + * Consumer callbacks are pre-compiled as separate classes and + * stored as fields on the main class. + */ +public final class LALClassGenerator { + + private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(0); + + private static final String PACKAGE_PREFIX = + "org.apache.skywalking.oap.log.analyzer.compiler.rt."; + + private static final String FILTER_SPEC = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec"; + private static final String BINDING = + "org.apache.skywalking.oap.log.analyzer.dsl.Binding"; + private static final String BINDING_PARSED = + "org.apache.skywalking.oap.log.analyzer.dsl.Binding.Parsed"; + + private final ClassPool classPool; + + public LALClassGenerator() { + this(ClassPool.getDefault()); + } + + public LALClassGenerator(final ClassPool classPool) { + this.classPool = classPool; + } + + /** + * Compiles a LAL DSL script into a LalExpression implementation. + */ + public LalExpression compile(final String dsl) throws Exception { + final LALScriptModel model = LALScriptParser.parse(dsl); + return compileFromModel(model); + } + + /** + * Compiles from a pre-parsed model. + */ + public LalExpression compileFromModel(final LALScriptModel model) throws Exception { + final String className = PACKAGE_PREFIX + "LalExpr_" + + CLASS_COUNTER.getAndIncrement(); + + // Phase 1: Collect all consumer info in traversal order + final List consumers = new ArrayList<>(); + collectConsumers(model.getStatements(), consumers); + + // Phase 2: Compile consumer classes + final List consumerInstances = new ArrayList<>(); + for (int i = 0; i < consumers.size(); i++) { + final String consumerName = className + "_C" + i; + final Object instance = compileConsumerClass( + consumerName, consumers.get(i)); + consumerInstances.add(instance); + } + + // Phase 3: Build main class with consumer fields + final CtClass ctClass = classPool.makeClass(className); + ctClass.addInterface(classPool.get( + "org.apache.skywalking.oap.log.analyzer.dsl.LalExpression")); + + for (int i = 0; i < consumers.size(); i++) { + ctClass.addField(CtField.make( + "public java.util.function.Consumer _consumer" + i + ";", + ctClass)); + } + + ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); + addHelperMethods(ctClass); + + // Phase 4: Generate execute method referencing consumer fields + final int[] counter = {0}; + final String executeBody = generateExecuteMethod(model, counter); + ctClass.addMethod(CtNewMethod.make(executeBody, ctClass)); + + final Class clazz = ctClass.toClass(LalExpressionPackageHolder.class); + ctClass.detach(); + final LalExpression instance = (LalExpression) clazz + .getDeclaredConstructor().newInstance(); + + // Phase 5: Wire consumer fields + for (int i = 0; i < consumerInstances.size(); i++) { + clazz.getField("_consumer" + i).set(instance, consumerInstances.get(i)); + } + + return instance; + } + + // ==================== Consumer info ==================== + + private static final class ConsumerInfo { + final String body; + final String castType; + final List subConsumers; + + ConsumerInfo(final String body, final String castType) { + this.body = body; + this.castType = castType; + this.subConsumers = new ArrayList<>(); + } + } + + // ==================== Phase 1: Collect consumers ==================== + + private void collectConsumers( + final List stmts, + final List consumers) { + for (final LALScriptModel.FilterStatement stmt : stmts) { + collectConsumerFromStatement(stmt, consumers); + } + } + + private void collectConsumerFromStatement( + final LALScriptModel.FilterStatement stmt, + final List consumers) { + if (stmt instanceof LALScriptModel.TextParser) { + final LALScriptModel.TextParser tp = (LALScriptModel.TextParser) stmt; + if (tp.getRegexpPattern() != null) { + final StringBuilder sb = new StringBuilder(); + sb.append(" _t.regexp(\"") + .append(escapeJava(tp.getRegexpPattern())) + .append("\");\n"); + consumers.add(new ConsumerInfo(sb.toString(), + "org.apache.skywalking.oap.log.analyzer.dsl" + + ".spec.parser.TextParserSpec")); + } + } else if (stmt instanceof LALScriptModel.JsonParser) { + if (((LALScriptModel.JsonParser) stmt).isAbortOnFailure()) { + consumers.add(new ConsumerInfo( + " _t.abortOnFailure();\n", + "org.apache.skywalking.oap.log.analyzer.dsl" + + ".spec.parser.JsonParserSpec")); + } + } else if (stmt instanceof LALScriptModel.YamlParser) { + if (((LALScriptModel.YamlParser) stmt).isAbortOnFailure()) { + consumers.add(new ConsumerInfo( + " _t.abortOnFailure();\n", + "org.apache.skywalking.oap.log.analyzer.dsl" + + ".spec.parser.YamlParserSpec")); + } + } else if (stmt instanceof LALScriptModel.ExtractorBlock) { + final LALScriptModel.ExtractorBlock block = + (LALScriptModel.ExtractorBlock) stmt; + final StringBuilder sb = new StringBuilder(); + generateExtractorStatementsFlat(sb, block.getStatements()); + consumers.add(new ConsumerInfo(sb.toString(), + "org.apache.skywalking.oap.log.analyzer.dsl" + + ".spec.extractor.ExtractorSpec")); + } else if (stmt instanceof LALScriptModel.SinkBlock) { + final LALScriptModel.SinkBlock sink = (LALScriptModel.SinkBlock) stmt; + if (!sink.getStatements().isEmpty()) { + final StringBuilder sb = new StringBuilder(); + generateSinkStatementsFlat(sb, sink.getStatements()); + consumers.add(new ConsumerInfo(sb.toString(), + "org.apache.skywalking.oap.log.analyzer.dsl" + + ".spec.sink.SinkSpec")); + } + } else if (stmt instanceof LALScriptModel.IfBlock) { + final LALScriptModel.IfBlock ifBlock = (LALScriptModel.IfBlock) stmt; + collectConsumers(ifBlock.getThenBranch(), consumers); + if (!ifBlock.getElseBranch().isEmpty()) { + collectConsumers(ifBlock.getElseBranch(), consumers); + } + } + } + + // ==================== Flat code for consumer bodies ==================== + + private void generateExtractorStatementsFlat( + final StringBuilder sb, + final List stmts) { + for (final LALScriptModel.ExtractorStatement stmt : stmts) { + if (stmt instanceof LALScriptModel.FieldAssignment) { + final LALScriptModel.FieldAssignment field = + (LALScriptModel.FieldAssignment) stmt; + sb.append(" _t.").append(field.getFieldType().name().toLowerCase()) + .append("("); + generateCastedValueAccess(sb, field.getValue(), + field.getCastType()); + if (field.getFormatPattern() != null) { + sb.append(", \"") + .append(escapeJava(field.getFormatPattern())) + .append("\""); + } + sb.append(");\n"); + } else if (stmt instanceof LALScriptModel.TagAssignment) { + final LALScriptModel.TagAssignment tag = + (LALScriptModel.TagAssignment) stmt; + if (tag.getTags().size() == 1) { + final Map.Entry entry = + tag.getTags().entrySet().iterator().next(); + sb.append(" _t.tag(\"") + .append(escapeJava(entry.getKey())).append("\", "); + generateCastedValueAccess(sb, entry.getValue().getValue(), + entry.getValue().getCastType()); + sb.append(");\n"); + } + } + } + } + + private void generateSinkStatementsFlat( + final StringBuilder sb, + final List stmts) { + for (final LALScriptModel.SinkStatement stmt : stmts) { + if (stmt instanceof LALScriptModel.EnforcerStatement) { + sb.append(" _t.enforcer();\n"); + } else if (stmt instanceof LALScriptModel.DropperStatement) { + sb.append(" _t.dropper();\n"); + } + } + } + + // ==================== Phase 2: Compile consumer classes ==================== + + private Object compileConsumerClass(final String className, + final ConsumerInfo info) throws Exception { + final CtClass ctClass = classPool.makeClass(className); + ctClass.addInterface(classPool.get("java.util.function.Consumer")); + ctClass.addInterface(classPool.get( + PACKAGE_PREFIX + "BindingAware")); + ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); + + ctClass.addField(CtField.make( + "private " + BINDING + " binding;", ctClass)); + + ctClass.addMethod(CtNewMethod.make( + "public void setBinding(" + BINDING + " b) {" + + " this.binding = b; }", ctClass)); + ctClass.addMethod(CtNewMethod.make( + "public " + BINDING + " getBinding() {" + + " return this.binding; }", ctClass)); + + addHelperMethods(ctClass); + + final String method = "public void accept(Object arg) {\n" + + " " + info.castType + " _t = (" + info.castType + ") arg;\n" + + info.body + + "}\n"; + ctClass.addMethod(CtNewMethod.make(method, ctClass)); + + final Class clazz = ctClass.toClass(LalExpressionPackageHolder.class); + ctClass.detach(); + return clazz.getDeclaredConstructor().newInstance(); + } + + // ==================== Phase 4: Generate execute method ==================== + + private String generateExecuteMethod(final LALScriptModel model, + final int[] counter) { + final StringBuilder sb = new StringBuilder(); + sb.append("public void execute(Object arg0, Object arg1) {\n"); + sb.append(" ").append(FILTER_SPEC).append(" filterSpec = (") + .append(FILTER_SPEC).append(") arg0;\n"); + sb.append(" ").append(BINDING).append(" binding = (") + .append(BINDING).append(") arg1;\n"); + + for (final LALScriptModel.FilterStatement stmt + : model.getStatements()) { + generateFilterStatement(sb, stmt, counter); + } + + sb.append("}\n"); + return sb.toString(); + } + + private void generateFilterStatement(final StringBuilder sb, + final LALScriptModel.FilterStatement stmt, + final int[] counter) { + if (stmt instanceof LALScriptModel.TextParser) { + final LALScriptModel.TextParser tp = (LALScriptModel.TextParser) stmt; + if (tp.getRegexpPattern() != null) { + emitConsumerCall(sb, "filterSpec.text", counter); + } else { + sb.append(" filterSpec.text();\n"); + } + } else if (stmt instanceof LALScriptModel.JsonParser) { + if (((LALScriptModel.JsonParser) stmt).isAbortOnFailure()) { + emitConsumerCall(sb, "filterSpec.json", counter); + } else { + sb.append(" filterSpec.json();\n"); + } + } else if (stmt instanceof LALScriptModel.YamlParser) { + if (((LALScriptModel.YamlParser) stmt).isAbortOnFailure()) { + emitConsumerCall(sb, "filterSpec.yaml", counter); + } else { + sb.append(" filterSpec.yaml();\n"); + } + } else if (stmt instanceof LALScriptModel.AbortStatement) { + sb.append(" filterSpec.abort();\n"); + } else if (stmt instanceof LALScriptModel.ExtractorBlock) { + emitConsumerCall(sb, "filterSpec.extractor", counter); + } else if (stmt instanceof LALScriptModel.SinkBlock) { + final LALScriptModel.SinkBlock sink = (LALScriptModel.SinkBlock) stmt; + if (sink.getStatements().isEmpty()) { + sb.append(" filterSpec.sink();\n"); + } else { + emitConsumerCall(sb, "filterSpec.sink", counter); + } + } else if (stmt instanceof LALScriptModel.IfBlock) { + generateIfBlock(sb, (LALScriptModel.IfBlock) stmt, counter); + } + } + + private void emitConsumerCall(final StringBuilder sb, + final String methodPrefix, + final int[] counter) { + final int idx = counter[0]++; + sb.append(" ((") + .append(PACKAGE_PREFIX).append("BindingAware) this._consumer") + .append(idx).append(").setBinding(binding);\n"); + sb.append(" ").append(methodPrefix) + .append("(this._consumer").append(idx).append(");\n"); + } + + private void generateIfBlock(final StringBuilder sb, + final LALScriptModel.IfBlock ifBlock, + final int[] counter) { + sb.append(" if ("); + generateCondition(sb, ifBlock.getCondition()); + sb.append(") {\n"); + for (final LALScriptModel.FilterStatement s : ifBlock.getThenBranch()) { + generateFilterStatement(sb, s, counter); + } + sb.append(" }\n"); + if (!ifBlock.getElseBranch().isEmpty()) { + sb.append(" else {\n"); + for (final LALScriptModel.FilterStatement s + : ifBlock.getElseBranch()) { + generateFilterStatement(sb, s, counter); + } + sb.append(" }\n"); + } + } + + // ==================== Helper methods ==================== + + private void addHelperMethods(final CtClass ctClass) throws Exception { + ctClass.addMethod(CtNewMethod.make( + "private static Object getAt(Object obj, String key) {" + + " if (obj == null) return null;" + + " if (obj instanceof " + BINDING_PARSED + ")" + + " return ((" + BINDING_PARSED + ") obj).getAt(key);" + + " if (obj instanceof java.util.Map)" + + " return ((java.util.Map) obj).get(key);" + + " return null;" + + "}", ctClass)); + + ctClass.addMethod(CtNewMethod.make( + "private static long toLong(Object obj) {" + + " if (obj instanceof Number) return ((Number) obj).longValue();" + + " if (obj instanceof String) return Long.parseLong((String) obj);" + + " return 0L;" + + "}", ctClass)); + + ctClass.addMethod(CtNewMethod.make( + "private static int toInt(Object obj) {" + + " if (obj instanceof Number) return ((Number) obj).intValue();" + + " if (obj instanceof String) return Integer.parseInt((String) obj);" + + " return 0;" + + "}", ctClass)); + + ctClass.addMethod(CtNewMethod.make( + "private static boolean toBool(Object obj) {" + + " if (obj instanceof Boolean) return ((Boolean) obj).booleanValue();" + + " if (obj instanceof String)" + + " return Boolean.parseBoolean((String) obj);" + + " return obj != null;" + + "}", ctClass)); + + ctClass.addMethod(CtNewMethod.make( + "private static boolean isTruthy(Object obj) {" + + " if (obj == null) return false;" + + " if (obj instanceof Boolean)" + + " return ((Boolean) obj).booleanValue();" + + " if (obj instanceof String)" + + " return !((String) obj).isEmpty();" + + " if (obj instanceof Number)" + + " return ((Number) obj).doubleValue() != 0;" + + " return true;" + + "}", ctClass)); + } + + // ==================== Conditions ==================== + + private void generateCondition(final StringBuilder sb, + final LALScriptModel.Condition cond) { + if (cond instanceof LALScriptModel.ComparisonCondition) { + final LALScriptModel.ComparisonCondition cc = + (LALScriptModel.ComparisonCondition) cond; + switch (cc.getOp()) { + case EQ: + sb.append("java.util.Objects.equals("); + generateValueAccessObj(sb, cc.getLeft(), cc.getLeftCast()); + sb.append(", "); + generateConditionValue(sb, cc.getRight()); + sb.append(")"); + break; + case NEQ: + sb.append("!java.util.Objects.equals("); + generateValueAccessObj(sb, cc.getLeft(), cc.getLeftCast()); + sb.append(", "); + generateConditionValue(sb, cc.getRight()); + sb.append(")"); + break; + case GT: + sb.append("toLong("); + generateValueAccessObj(sb, cc.getLeft(), null); + sb.append(") > "); + generateConditionValueNumeric(sb, cc.getRight()); + break; + case LT: + sb.append("toLong("); + generateValueAccessObj(sb, cc.getLeft(), null); + sb.append(") < "); + generateConditionValueNumeric(sb, cc.getRight()); + break; + case GTE: + sb.append("toLong("); + generateValueAccessObj(sb, cc.getLeft(), null); + sb.append(") >= "); + generateConditionValueNumeric(sb, cc.getRight()); + break; + case LTE: + sb.append("toLong("); + generateValueAccessObj(sb, cc.getLeft(), null); + sb.append(") <= "); + generateConditionValueNumeric(sb, cc.getRight()); + break; + default: + break; + } + } else if (cond instanceof LALScriptModel.LogicalCondition) { + final LALScriptModel.LogicalCondition lc = + (LALScriptModel.LogicalCondition) cond; + sb.append("("); + generateCondition(sb, lc.getLeft()); + sb.append(lc.getOp() == LALScriptModel.LogicalOp.AND + ? " && " : " || "); + generateCondition(sb, lc.getRight()); + sb.append(")"); + } else if (cond instanceof LALScriptModel.NotCondition) { + sb.append("!("); + generateCondition(sb, + ((LALScriptModel.NotCondition) cond).getInner()); + sb.append(")"); + } else if (cond instanceof LALScriptModel.ExprCondition) { + sb.append("isTruthy("); + generateValueAccessObj(sb, + ((LALScriptModel.ExprCondition) cond).getExpr(), + ((LALScriptModel.ExprCondition) cond).getCastType()); + sb.append(")"); + } + } + + private void generateConditionValue(final StringBuilder sb, + final LALScriptModel.ConditionValue cv) { + if (cv instanceof LALScriptModel.StringConditionValue) { + sb.append('"') + .append(escapeJava( + ((LALScriptModel.StringConditionValue) cv).getValue())) + .append('"'); + } else if (cv instanceof LALScriptModel.NumberConditionValue) { + final double val = + ((LALScriptModel.NumberConditionValue) cv).getValue(); + sb.append("Long.valueOf(").append((long) val).append("L)"); + } else if (cv instanceof LALScriptModel.BoolConditionValue) { + sb.append("Boolean.valueOf(") + .append(((LALScriptModel.BoolConditionValue) cv).isValue()) + .append(")"); + } else if (cv instanceof LALScriptModel.NullConditionValue) { + sb.append("null"); + } else if (cv instanceof LALScriptModel.ValueAccessConditionValue) { + generateValueAccessObj(sb, + ((LALScriptModel.ValueAccessConditionValue) cv).getValue(), + null); + } + } + + private void generateConditionValueNumeric( + final StringBuilder sb, + final LALScriptModel.ConditionValue cv) { + if (cv instanceof LALScriptModel.NumberConditionValue) { + sb.append((long) ((LALScriptModel.NumberConditionValue) cv) + .getValue()).append("L"); + } else if (cv instanceof LALScriptModel.ValueAccessConditionValue) { + sb.append("toLong("); + generateValueAccessObj(sb, + ((LALScriptModel.ValueAccessConditionValue) cv).getValue(), + null); + sb.append(")"); + } else { + sb.append("0L"); + } + } + + // ==================== Value access ==================== + + private void generateCastedValueAccess(final StringBuilder sb, + final LALScriptModel.ValueAccess value, + final String castType) { + if ("String".equals(castType)) { + sb.append("String.valueOf("); + generateValueAccess(sb, value); + sb.append(")"); + } else if ("Long".equals(castType)) { + sb.append("toLong("); + generateValueAccess(sb, value); + sb.append(")"); + } else if ("Integer".equals(castType)) { + sb.append("toInt("); + generateValueAccess(sb, value); + sb.append(")"); + } else if ("Boolean".equals(castType)) { + sb.append("toBool("); + generateValueAccess(sb, value); + sb.append(")"); + } else { + generateValueAccess(sb, value); + } + } + + private void generateValueAccessObj(final StringBuilder sb, + final LALScriptModel.ValueAccess value, + final String castType) { + if ("String".equals(castType)) { + sb.append("String.valueOf("); + generateValueAccess(sb, value); + sb.append(")"); + } else { + generateValueAccess(sb, value); + } + } + + private void generateValueAccess(final StringBuilder sb, + final LALScriptModel.ValueAccess value) { + String current; + if (value.isParsedRef()) { + current = "binding.parsed()"; + } else if (value.isLogRef()) { + current = "binding.log()"; + } else { + final List segs = value.getChain(); + if (segs.isEmpty()) { + sb.append("null"); + return; + } + current = "binding.parsed()"; + } + + final List chain = value.getChain(); + if (chain.isEmpty()) { + sb.append(current); + return; + } + + for (int i = 0; i < chain.size(); i++) { + final LALScriptModel.ValueAccessSegment seg = chain.get(i); + if (seg instanceof LALScriptModel.FieldSegment) { + final String name = + ((LALScriptModel.FieldSegment) seg).getName(); + current = "getAt(" + current + ", \"" + + escapeJava(name) + "\")"; + } else if (seg instanceof LALScriptModel.MethodSegment) { + final LALScriptModel.MethodSegment ms = + (LALScriptModel.MethodSegment) seg; + current = current + "." + ms.getName() + "()"; + } + } + sb.append(current); + } + + // ==================== Utilities ==================== + + private static String escapeJava(final String s) { + return s.replace("\\", "\\\\") + .replace("\"", "\\\"") + .replace("\n", "\\n") + .replace("\r", "\\r") + .replace("\t", "\\t"); + } + + /** + * Generates the Java source body of the execute method for + * debugging/testing. + */ + public String generateSource(final String dsl) { + final LALScriptModel model = LALScriptParser.parse(dsl); + final int[] counter = {0}; + return generateExecuteMethod(model, counter); + } +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java new file mode 100644 index 000000000000..cc3d1d3df04e --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java @@ -0,0 +1,459 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import lombok.Getter; + +/** + * Immutable AST model for LAL (Log Analysis Language) scripts. + * + *

Represents parsed scripts like: + * 

+ *   filter {
+ *     json {}
+ *     extractor { service parsed.service as String }
+ *     sink { sampler { rateLimit("id") { rpm 6000 } } }
+ *   }
+ *
+ */ +public final class LALScriptModel { + + @Getter + private final List statements; + + public LALScriptModel(final List statements) { + this.statements = Collections.unmodifiableList(statements); + } + + // ==================== Filter statements ==================== + + public interface FilterStatement { + } + + // ==================== Parser blocks ==================== + + @Getter + public static final class TextParser implements FilterStatement { + private final String regexpPattern; + private final boolean abortOnFailure; + + public TextParser(final String regexpPattern, final boolean abortOnFailure) { + this.regexpPattern = regexpPattern; + this.abortOnFailure = abortOnFailure; + } + } + + @Getter + public static final class JsonParser implements FilterStatement { + private final boolean abortOnFailure; + + public JsonParser(final boolean abortOnFailure) { + this.abortOnFailure = abortOnFailure; + } + } + + @Getter + public static final class YamlParser implements FilterStatement { + private final boolean abortOnFailure; + + public YamlParser(final boolean abortOnFailure) { + this.abortOnFailure = abortOnFailure; + } + } + + public static final class AbortStatement implements FilterStatement { + } + + // ==================== Extractor block ==================== + + @Getter + public static final class ExtractorBlock implements FilterStatement { + private final List statements; + + public ExtractorBlock(final List statements) { + this.statements = Collections.unmodifiableList(statements); + } + } + + public interface ExtractorStatement { + } + + @Getter + public static final class FieldAssignment implements ExtractorStatement, FilterStatement { + private final FieldType fieldType; + private final ValueAccess value; + private final String castType; + private final String formatPattern; + + public FieldAssignment(final FieldType fieldType, + final ValueAccess value, + final String castType, + final String formatPattern) { + this.fieldType = fieldType; + this.value = value; + this.castType = castType; + this.formatPattern = formatPattern; + } + } + + public enum FieldType { + SERVICE, INSTANCE, ENDPOINT, LAYER, + TRACE_ID, SEGMENT_ID, SPAN_ID, TIMESTAMP + } + + @Getter + public static final class TagAssignment implements ExtractorStatement, FilterStatement { + private final Map tags; + + public TagAssignment(final Map tags) { + this.tags = Collections.unmodifiableMap(tags); + } + } + + @Getter + public static final class TagValue { + private final ValueAccess value; + private final String castType; + + public TagValue(final ValueAccess value, final String castType) { + this.value = value; + this.castType = castType; + } + } + + @Getter + public static final class MetricsBlock implements ExtractorStatement, FilterStatement { + private final String name; + private final ValueAccess timestampValue; + private final String timestampCast; + private final Map labels; + private final ValueAccess value; + private final String valueCast; + + public MetricsBlock(final String name, + final ValueAccess timestampValue, + final String timestampCast, + final Map labels, + final ValueAccess value, + final String valueCast) { + this.name = name; + this.timestampValue = timestampValue; + this.timestampCast = timestampCast; + this.labels = labels != null ? Collections.unmodifiableMap(labels) : Collections.emptyMap(); + this.value = value; + this.valueCast = valueCast; + } + } + + @Getter + public static final class SlowSqlBlock implements ExtractorStatement, FilterStatement { + private final ValueAccess id; + private final String idCast; + private final ValueAccess statement; + private final String statementCast; + private final ValueAccess latency; + private final String latencyCast; + + public SlowSqlBlock(final ValueAccess id, final String idCast, + final ValueAccess statement, final String statementCast, + final ValueAccess latency, final String latencyCast) { + this.id = id; + this.idCast = idCast; + this.statement = statement; + this.statementCast = statementCast; + this.latency = latency; + this.latencyCast = latencyCast; + } + } + + @Getter + public static final class SampledTraceBlock implements ExtractorStatement, FilterStatement { + private final List statements; + + public SampledTraceBlock(final List statements) { + this.statements = Collections.unmodifiableList(statements); + } + } + + public interface SampledTraceStatement { + } + + @Getter + public static final class SampledTraceField implements SampledTraceStatement { + private final SampledTraceFieldType fieldType; + private final ValueAccess value; + private final String castType; + + public SampledTraceField(final SampledTraceFieldType fieldType, + final ValueAccess value, + final String castType) { + this.fieldType = fieldType; + this.value = value; + this.castType = castType; + } + } + + public enum SampledTraceFieldType { + LATENCY, URI, REASON, PROCESS_ID, DEST_PROCESS_ID, + DETECT_POINT, COMPONENT_ID, REPORT_SERVICE + } + + // ==================== Sink block ==================== + + @Getter + public static final class SinkBlock implements FilterStatement { + private final List statements; + + public SinkBlock(final List statements) { + this.statements = Collections.unmodifiableList(statements); + } + } + + public interface SinkStatement { + } + + @Getter + public static final class SamplerBlock implements SinkStatement, FilterStatement { + private final List contents; + + public SamplerBlock(final List contents) { + this.contents = Collections.unmodifiableList(contents); + } + } + + public interface SamplerContent { + } + + @Getter + public static final class RateLimitBlock implements SamplerContent { + private final String id; + private final long rpm; + + public RateLimitBlock(final String id, final long rpm) { + this.id = id; + this.rpm = rpm; + } + } + + public static final class EnforcerStatement implements SinkStatement, FilterStatement { + } + + public static final class DropperStatement implements SinkStatement, FilterStatement { + } + + // ==================== Control flow ==================== + + @Getter + public static final class IfBlock implements FilterStatement, ExtractorStatement, + SinkStatement, SampledTraceStatement, SamplerContent { + private final Condition condition; + private final List thenBranch; + private final List elseBranch; + + public IfBlock(final Condition condition, + final List thenBranch, + final List elseBranch) { + this.condition = condition; + this.thenBranch = Collections.unmodifiableList(thenBranch); + this.elseBranch = elseBranch != null + ? Collections.unmodifiableList(elseBranch) : Collections.emptyList(); + } + } + + // ==================== Conditions ==================== + + public interface Condition { + } + + @Getter + public static final class ComparisonCondition implements Condition { + private final ValueAccess left; + private final String leftCast; + private final CompareOp op; + private final ConditionValue right; + + public ComparisonCondition(final ValueAccess left, + final String leftCast, + final CompareOp op, + final ConditionValue right) { + this.left = left; + this.leftCast = leftCast; + this.op = op; + this.right = right; + } + } + + @Getter + public static final class LogicalCondition implements Condition { + private final Condition left; + private final LogicalOp op; + private final Condition right; + + public LogicalCondition(final Condition left, final LogicalOp op, final Condition right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + @Getter + public static final class NotCondition implements Condition { + private final Condition inner; + + public NotCondition(final Condition inner) { + this.inner = inner; + } + } + + @Getter + public static final class ExprCondition implements Condition { + private final ValueAccess expr; + private final String castType; + + public ExprCondition(final ValueAccess expr, final String castType) { + this.expr = expr; + this.castType = castType; + } + } + + // ==================== Value access ==================== + + @Getter + public static final class ValueAccess { + private final List segments; + private final boolean parsedRef; + private final boolean logRef; + private final List chain; + + public ValueAccess(final List segments, + final boolean parsedRef, + final boolean logRef, + final List chain) { + this.segments = Collections.unmodifiableList(segments); + this.parsedRef = parsedRef; + this.logRef = logRef; + this.chain = chain != null + ? Collections.unmodifiableList(chain) : Collections.emptyList(); + } + + public String toPathString() { + return String.join(".", segments); + } + } + + public interface ValueAccessSegment { + } + + @Getter + public static final class FieldSegment implements ValueAccessSegment { + private final String name; + private final boolean safeNav; + + public FieldSegment(final String name, final boolean safeNav) { + this.name = name; + this.safeNav = safeNav; + } + } + + @Getter + public static final class MethodSegment implements ValueAccessSegment { + private final String name; + private final List arguments; + private final boolean safeNav; + + public MethodSegment(final String name, final List arguments, final boolean safeNav) { + this.name = name; + this.arguments = Collections.unmodifiableList(arguments); + this.safeNav = safeNav; + } + } + + // ==================== Condition values ==================== + + public interface ConditionValue { + } + + @Getter + public static final class StringConditionValue implements ConditionValue { + private final String value; + + public StringConditionValue(final String value) { + this.value = value; + } + } + + @Getter + public static final class NumberConditionValue implements ConditionValue { + private final double value; + + public NumberConditionValue(final double value) { + this.value = value; + } + } + + @Getter + public static final class BoolConditionValue implements ConditionValue { + private final boolean value; + + public BoolConditionValue(final boolean value) { + this.value = value; + } + } + + public static final class NullConditionValue implements ConditionValue { + } + + @Getter + public static final class ValueAccessConditionValue implements ConditionValue { + private final ValueAccess value; + private final String castType; + + public ValueAccessConditionValue(final ValueAccess value, final String castType) { + this.value = value; + this.castType = castType; + } + } + + @Getter + public static final class FunctionCallConditionValue implements ConditionValue { + private final String functionName; + private final List arguments; + + public FunctionCallConditionValue(final String functionName, final List arguments) { + this.functionName = functionName; + this.arguments = Collections.unmodifiableList(arguments); + } + } + + // ==================== Enums ==================== + + public enum CompareOp { + EQ, NEQ, GT, LT, GTE, LTE + } + + public enum LogicalOp { + AND, OR + } + + private LALScriptModel() { + this.statements = Collections.emptyList(); + } +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java new file mode 100644 index 000000000000..128f32557362 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java @@ -0,0 +1,687 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import org.antlr.v4.runtime.BaseErrorListener; +import org.antlr.v4.runtime.CharStreams; +import org.antlr.v4.runtime.CommonTokenStream; +import org.antlr.v4.runtime.RecognitionException; +import org.antlr.v4.runtime.Recognizer; +import org.apache.skywalking.lal.rt.grammar.LALLexer; +import org.apache.skywalking.lal.rt.grammar.LALParser; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.AbortStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.CompareOp; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ComparisonCondition; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.Condition; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.DropperStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.EnforcerStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ExprCondition; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ExtractorBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ExtractorStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.FieldAssignment; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.FieldSegment; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.FieldType; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.FilterStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.IfBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.JsonParser; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.LogicalCondition; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.LogicalOp; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.MetricsBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.NotCondition; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.NullConditionValue; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.NumberConditionValue; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.RateLimitBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.SamplerBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.SamplerContent; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.SinkBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.SinkStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.SlowSqlBlock; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.StringConditionValue; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.TagAssignment; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.TagValue; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.TextParser; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ValueAccess; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ValueAccessConditionValue; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ValueAccessSegment; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.YamlParser; + +/** + * Facade: parses LAL DSL script strings into {@link LALScriptModel}. + * + * 
+ *   LALScriptModel model = LALScriptParser.parse(
+ *       "filter { json {} extractor { service parsed.service as String } sink {} }");
+ *
+ */ +public final class LALScriptParser { + + private LALScriptParser() { + } + + public static LALScriptModel parse(final String dsl) { + final LALLexer lexer = new LALLexer(CharStreams.fromString(dsl)); + final CommonTokenStream tokens = new CommonTokenStream(lexer); + final LALParser parser = new LALParser(tokens); + + final List errors = new ArrayList<>(); + parser.removeErrorListeners(); + parser.addErrorListener(new BaseErrorListener() { + @Override + public void syntaxError(final Recognizer recognizer, + final Object offendingSymbol, + final int line, + final int charPositionInLine, + final String msg, + final RecognitionException e) { + errors.add(line + ":" + charPositionInLine + " " + msg); + } + }); + + final LALParser.RootContext tree = parser.root(); + if (!errors.isEmpty()) { + throw new IllegalArgumentException( + "LAL script parsing failed: " + String.join("; ", errors) + + " in script: " + truncate(dsl, 200)); + } + + final List stmts = visitFilterContent( + tree.filterBlock().filterContent()); + return new LALScriptModel(stmts); + } + + // ==================== Filter content ==================== + + private static List visitFilterContent( + final LALParser.FilterContentContext ctx) { + final List stmts = new ArrayList<>(); + for (final LALParser.FilterStatementContext fsc : ctx.filterStatement()) { + stmts.add(visitFilterStatement(fsc)); + } + return stmts; + } + + private static FilterStatement visitFilterStatement( + final LALParser.FilterStatementContext ctx) { + if (ctx.parserBlock() != null) { + return visitParserBlock(ctx.parserBlock()); + } + if (ctx.extractorBlock() != null) { + return visitExtractorBlock(ctx.extractorBlock()); + } + if (ctx.sinkBlock() != null) { + return visitSinkBlock(ctx.sinkBlock()); + } + if (ctx.abortBlock() != null) { + return new AbortStatement(); + } + // ifStatement + return visitIfStatement(ctx.ifStatement()); + } + + // ==================== Parser blocks ==================== + + private static FilterStatement visitParserBlock(final LALParser.ParserBlockContext ctx) { + if (ctx.textBlock() != null) { + String pattern = null; + boolean abortOnFail = false; + if (ctx.textBlock().textContent() != null) { + for (final LALParser.RegexpStatementContext regCtx : + ctx.textBlock().textContent().regexpStatement()) { + pattern = stripQuotes(regCtx.regexpPattern().getText()); + } + for (final LALParser.AbortOnFailureStatementContext abfCtx : + ctx.textBlock().textContent().abortOnFailureStatement()) { + abortOnFail = "true".equals(abfCtx.boolValue().getText()); + } + } + return new TextParser(pattern, abortOnFail); + } + if (ctx.jsonBlock() != null) { + boolean abortOnFail = false; + if (ctx.jsonBlock().jsonContent() != null + && ctx.jsonBlock().jsonContent().abortOnFailureStatement() != null) { + abortOnFail = "true".equals( + ctx.jsonBlock().jsonContent().abortOnFailureStatement() + .boolValue().getText()); + } + return new JsonParser(abortOnFail); + } + // yaml + boolean abortOnFail = false; + if (ctx.yamlBlock().yamlContent() != null + && ctx.yamlBlock().yamlContent().abortOnFailureStatement() != null) { + abortOnFail = "true".equals( + ctx.yamlBlock().yamlContent().abortOnFailureStatement() + .boolValue().getText()); + } + return new YamlParser(abortOnFail); + } + + // ==================== Extractor block ==================== + + private static ExtractorBlock visitExtractorBlock( + final LALParser.ExtractorBlockContext ctx) { + final List stmts = new ArrayList<>(); + for (final LALParser.ExtractorStatementContext esc : ctx.extractorContent().extractorStatement()) { + stmts.add(visitExtractorStatement(esc)); + } + return new ExtractorBlock(stmts); + } + + private static ExtractorStatement visitExtractorStatement( + final LALParser.ExtractorStatementContext ctx) { + if (ctx.serviceStatement() != null) { + return visitFieldAssignment(FieldType.SERVICE, ctx.serviceStatement().valueAccess(), + ctx.serviceStatement().typeCast()); + } + if (ctx.instanceStatement() != null) { + return visitFieldAssignment(FieldType.INSTANCE, ctx.instanceStatement().valueAccess(), + ctx.instanceStatement().typeCast()); + } + if (ctx.endpointStatement() != null) { + return visitFieldAssignment(FieldType.ENDPOINT, ctx.endpointStatement().valueAccess(), + ctx.endpointStatement().typeCast()); + } + if (ctx.layerStatement() != null) { + return visitFieldAssignment(FieldType.LAYER, ctx.layerStatement().valueAccess(), + ctx.layerStatement().typeCast()); + } + if (ctx.traceIdStatement() != null) { + return visitFieldAssignment(FieldType.TRACE_ID, ctx.traceIdStatement().valueAccess(), + ctx.traceIdStatement().typeCast()); + } + if (ctx.timestampStatement() != null) { + final ValueAccess va = visitValueAccess(ctx.timestampStatement().valueAccess()); + final String cast = ctx.timestampStatement().typeCast() != null + ? extractCastType(ctx.timestampStatement().typeCast()) : null; + String format = null; + if (ctx.timestampStatement().STRING() != null) { + format = stripQuotes(ctx.timestampStatement().STRING().getText()); + } + return new FieldAssignment(FieldType.TIMESTAMP, va, cast, format); + } + if (ctx.tagStatement() != null) { + return visitTagStatement(ctx.tagStatement()); + } + if (ctx.metricsBlock() != null) { + return visitMetricsBlock(ctx.metricsBlock()); + } + if (ctx.slowSqlBlock() != null) { + return visitSlowSqlBlock(ctx.slowSqlBlock()); + } + if (ctx.sampledTraceBlock() != null) { + return visitSampledTraceBlock(ctx.sampledTraceBlock()); + } + // ifStatement + return (ExtractorStatement) visitIfStatement(ctx.ifStatement()); + } + + private static FieldAssignment visitFieldAssignment( + final FieldType type, + final LALParser.ValueAccessContext vaCtx, + final LALParser.TypeCastContext tcCtx) { + final ValueAccess va = visitValueAccess(vaCtx); + final String cast = tcCtx != null ? extractCastType(tcCtx) : null; + return new FieldAssignment(type, va, cast, null); + } + + // ==================== Tag statement ==================== + + private static TagAssignment visitTagStatement(final LALParser.TagStatementContext ctx) { + final Map tags = new LinkedHashMap<>(); + if (ctx.tagMap() != null) { + for (int i = 0; i < ctx.tagMap().anyIdentifier().size(); i++) { + final String key = ctx.tagMap().anyIdentifier(i).getText(); + final ValueAccess va = visitValueAccess(ctx.tagMap().valueAccess(i)); + final String cast = ctx.tagMap().typeCast(i) != null + ? extractCastType(ctx.tagMap().typeCast(i)) : null; + tags.put(key, new TagValue(va, cast)); + } + } else if (ctx.STRING() != null) { + final String key = stripQuotes(ctx.STRING().getText()); + final ValueAccess va = visitValueAccess(ctx.valueAccess()); + final String cast = ctx.typeCast() != null ? extractCastType(ctx.typeCast()) : null; + tags.put(key, new TagValue(va, cast)); + } + return new TagAssignment(tags); + } + + // ==================== Metrics block ==================== + + private static MetricsBlock visitMetricsBlock(final LALParser.MetricsBlockContext ctx) { + String name = null; + ValueAccess timestampValue = null; + String timestampCast = null; + final Map labels = new LinkedHashMap<>(); + ValueAccess value = null; + String valueCast = null; + + for (final LALParser.MetricsStatementContext msc : ctx.metricsContent().metricsStatement()) { + if (msc.metricsNameStatement() != null) { + name = resolveValueAsString(msc.metricsNameStatement().valueAccess()); + } + if (msc.metricsTimestampStatement() != null) { + timestampValue = visitValueAccess(msc.metricsTimestampStatement().valueAccess()); + timestampCast = msc.metricsTimestampStatement().typeCast() != null + ? extractCastType(msc.metricsTimestampStatement().typeCast()) : null; + } + if (msc.metricsLabelsStatement() != null) { + for (final LALParser.LabelEntryContext lec : + msc.metricsLabelsStatement().labelMap().labelEntry()) { + final String key = lec.anyIdentifier().getText(); + final ValueAccess va = visitValueAccess(lec.valueAccess()); + final String cast = lec.typeCast() != null + ? extractCastType(lec.typeCast()) : null; + labels.put(key, new TagValue(va, cast)); + } + } + if (msc.metricsValueStatement() != null) { + value = visitValueAccess(msc.metricsValueStatement().valueAccess()); + valueCast = msc.metricsValueStatement().typeCast() != null + ? extractCastType(msc.metricsValueStatement().typeCast()) : null; + } + } + + return new MetricsBlock(name, timestampValue, timestampCast, labels, value, valueCast); + } + + // ==================== Slow SQL block ==================== + + private static SlowSqlBlock visitSlowSqlBlock(final LALParser.SlowSqlBlockContext ctx) { + ValueAccess id = null; + String idCast = null; + ValueAccess statement = null; + String statementCast = null; + ValueAccess latency = null; + String latencyCast = null; + + for (final LALParser.SlowSqlStatementContext ssc : + ctx.slowSqlContent().slowSqlStatement()) { + if (ssc.slowSqlIdStatement() != null) { + id = visitValueAccess(ssc.slowSqlIdStatement().valueAccess()); + idCast = ssc.slowSqlIdStatement().typeCast() != null + ? extractCastType(ssc.slowSqlIdStatement().typeCast()) : null; + } + if (ssc.slowSqlStatementStatement() != null) { + statement = visitValueAccess(ssc.slowSqlStatementStatement().valueAccess()); + statementCast = ssc.slowSqlStatementStatement().typeCast() != null + ? extractCastType(ssc.slowSqlStatementStatement().typeCast()) : null; + } + if (ssc.slowSqlLatencyStatement() != null) { + latency = visitValueAccess(ssc.slowSqlLatencyStatement().valueAccess()); + latencyCast = ssc.slowSqlLatencyStatement().typeCast() != null + ? extractCastType(ssc.slowSqlLatencyStatement().typeCast()) : null; + } + } + + return new SlowSqlBlock(id, idCast, statement, statementCast, latency, latencyCast); + } + + // ==================== Sampled trace block ==================== + + private static LALScriptModel.SampledTraceBlock visitSampledTraceBlock( + final LALParser.SampledTraceBlockContext ctx) { + final List stmts = new ArrayList<>(); + for (final LALParser.SampledTraceStatementContext stc : + ctx.sampledTraceContent().sampledTraceStatement()) { + if (stc.ifStatement() != null) { + stmts.add((LALScriptModel.SampledTraceStatement) visitIfStatement( + stc.ifStatement())); + } else { + stmts.add(visitSampledTraceField(stc)); + } + } + return new LALScriptModel.SampledTraceBlock(stmts); + } + + private static LALScriptModel.SampledTraceField visitSampledTraceField( + final LALParser.SampledTraceStatementContext ctx) { + if (ctx.sampledTraceLatencyStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.LATENCY, + ctx.sampledTraceLatencyStatement().valueAccess(), + ctx.sampledTraceLatencyStatement().typeCast()); + } + if (ctx.sampledTraceUriStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.URI, + ctx.sampledTraceUriStatement().valueAccess(), + ctx.sampledTraceUriStatement().typeCast()); + } + if (ctx.sampledTraceReasonStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.REASON, + ctx.sampledTraceReasonStatement().valueAccess(), + ctx.sampledTraceReasonStatement().typeCast()); + } + if (ctx.sampledTraceProcessIdStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.PROCESS_ID, + ctx.sampledTraceProcessIdStatement().valueAccess(), + ctx.sampledTraceProcessIdStatement().typeCast()); + } + if (ctx.sampledTraceDestProcessIdStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.DEST_PROCESS_ID, + ctx.sampledTraceDestProcessIdStatement().valueAccess(), + ctx.sampledTraceDestProcessIdStatement().typeCast()); + } + if (ctx.sampledTraceDetectPointStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.DETECT_POINT, + ctx.sampledTraceDetectPointStatement().valueAccess(), + ctx.sampledTraceDetectPointStatement().typeCast()); + } + if (ctx.sampledTraceComponentIdStatement() != null) { + return makeSampledField(LALScriptModel.SampledTraceFieldType.COMPONENT_ID, + ctx.sampledTraceComponentIdStatement().valueAccess(), + ctx.sampledTraceComponentIdStatement().typeCast()); + } + // reportService + return makeSampledField(LALScriptModel.SampledTraceFieldType.REPORT_SERVICE, + ctx.reportServiceStatement().valueAccess(), + ctx.reportServiceStatement().typeCast()); + } + + private static LALScriptModel.SampledTraceField makeSampledField( + final LALScriptModel.SampledTraceFieldType type, + final LALParser.ValueAccessContext vaCtx, + final LALParser.TypeCastContext tcCtx) { + return new LALScriptModel.SampledTraceField( + type, visitValueAccess(vaCtx), + tcCtx != null ? extractCastType(tcCtx) : null); + } + + // ==================== Sink block ==================== + + private static SinkBlock visitSinkBlock(final LALParser.SinkBlockContext ctx) { + final List stmts = new ArrayList<>(); + for (final LALParser.SinkStatementContext ssc : ctx.sinkContent().sinkStatement()) { + if (ssc.samplerBlock() != null) { + stmts.add(visitSamplerBlock(ssc.samplerBlock())); + } else if (ssc.enforcerStatement() != null) { + stmts.add(new EnforcerStatement()); + } else if (ssc.dropperStatement() != null) { + stmts.add(new DropperStatement()); + } else { + stmts.add((SinkStatement) visitIfStatement(ssc.ifStatement())); + } + } + return new SinkBlock(stmts); + } + + private static SamplerBlock visitSamplerBlock(final LALParser.SamplerBlockContext ctx) { + final List contents = new ArrayList<>(); + for (final LALParser.RateLimitBlockContext rlc : ctx.samplerContent().rateLimitBlock()) { + final String id = stripQuotes(rlc.rateLimitId().getText()); + final long rpm = Long.parseLong(rlc.rateLimitContent().NUMBER().getText()); + contents.add(new RateLimitBlock(id, rpm)); + } + for (final LALParser.IfStatementContext isc : ctx.samplerContent().ifStatement()) { + contents.add((SamplerContent) visitIfStatement(isc)); + } + return new SamplerBlock(contents); + } + + // ==================== If statement ==================== + + private static IfBlock visitIfStatement(final LALParser.IfStatementContext ctx) { + final Condition condition = visitCondition(ctx.condition(0)); + final List thenBranch = visitIfBody(ctx.ifBody(0)); + + List elseBranch = null; + // Handle else-if and else branches + final int bodyCount = ctx.ifBody().size(); + if (bodyCount > 1) { + elseBranch = visitIfBody(ctx.ifBody(bodyCount - 1)); + } + + return new IfBlock(condition, thenBranch, elseBranch); + } + + private static List visitIfBody(final LALParser.IfBodyContext ctx) { + final List stmts = new ArrayList<>(); + for (final LALParser.FilterStatementContext fsc : ctx.filterStatement()) { + stmts.add(visitFilterStatement(fsc)); + } + for (final LALParser.ExtractorStatementContext esc : ctx.extractorStatement()) { + stmts.add((FilterStatement) visitExtractorStatement(esc)); + } + for (final LALParser.SinkStatementContext ssc : ctx.sinkStatement()) { + if (ssc.samplerBlock() != null) { + stmts.add((FilterStatement) visitSamplerBlock(ssc.samplerBlock())); + } else if (ssc.enforcerStatement() != null) { + stmts.add((FilterStatement) new EnforcerStatement()); + } else if (ssc.dropperStatement() != null) { + stmts.add((FilterStatement) new DropperStatement()); + } + } + return stmts; + } + + // ==================== Conditions ==================== + + private static Condition visitCondition(final LALParser.ConditionContext ctx) { + if (ctx instanceof LALParser.CondAndContext) { + final LALParser.CondAndContext and = (LALParser.CondAndContext) ctx; + return new LogicalCondition( + visitCondition(and.condition(0)), + LogicalOp.AND, + visitCondition(and.condition(1))); + } + if (ctx instanceof LALParser.CondOrContext) { + final LALParser.CondOrContext or = (LALParser.CondOrContext) ctx; + return new LogicalCondition( + visitCondition(or.condition(0)), + LogicalOp.OR, + visitCondition(or.condition(1))); + } + if (ctx instanceof LALParser.CondNotContext) { + return new NotCondition( + visitCondition(((LALParser.CondNotContext) ctx).condition())); + } + if (ctx instanceof LALParser.CondParenContext) { + return visitCondition(((LALParser.CondParenContext) ctx).condition()); + } + if (ctx instanceof LALParser.CondEqContext) { + final LALParser.CondEqContext eq = (LALParser.CondEqContext) ctx; + return makeComparison(eq.conditionExpr(0), CompareOp.EQ, eq.conditionExpr(1)); + } + if (ctx instanceof LALParser.CondNeqContext) { + final LALParser.CondNeqContext neq = (LALParser.CondNeqContext) ctx; + return makeComparison(neq.conditionExpr(0), CompareOp.NEQ, neq.conditionExpr(1)); + } + if (ctx instanceof LALParser.CondGtContext) { + final LALParser.CondGtContext gt = (LALParser.CondGtContext) ctx; + return makeComparison(gt.conditionExpr(0), CompareOp.GT, gt.conditionExpr(1)); + } + if (ctx instanceof LALParser.CondLtContext) { + final LALParser.CondLtContext lt = (LALParser.CondLtContext) ctx; + return makeComparison(lt.conditionExpr(0), CompareOp.LT, lt.conditionExpr(1)); + } + if (ctx instanceof LALParser.CondGteContext) { + final LALParser.CondGteContext gte = (LALParser.CondGteContext) ctx; + return makeComparison(gte.conditionExpr(0), CompareOp.GTE, gte.conditionExpr(1)); + } + if (ctx instanceof LALParser.CondLteContext) { + final LALParser.CondLteContext lte = (LALParser.CondLteContext) ctx; + return makeComparison(lte.conditionExpr(0), CompareOp.LTE, lte.conditionExpr(1)); + } + // condSingle + final LALParser.CondSingleContext single = (LALParser.CondSingleContext) ctx; + return visitConditionExprAsCondition(single.conditionExpr()); + } + + private static Condition makeComparison( + final LALParser.ConditionExprContext leftCtx, + final CompareOp op, + final LALParser.ConditionExprContext rightCtx) { + if (leftCtx instanceof LALParser.CondValueAccessContext) { + final LALParser.CondValueAccessContext lva = + (LALParser.CondValueAccessContext) leftCtx; + final ValueAccess left = visitValueAccess(lva.valueAccess()); + final String leftCast = lva.typeCast() != null + ? extractCastType(lva.typeCast()) : null; + return new ComparisonCondition(left, leftCast, op, + visitConditionExprAsValue(rightCtx)); + } + // For function calls and other forms, wrap as expression condition + return new ExprCondition( + new ValueAccess(List.of(leftCtx.getText()), false, false, List.of()), null); + } + + private static LALScriptModel.ConditionValue visitConditionExprAsValue( + final LALParser.ConditionExprContext ctx) { + if (ctx instanceof LALParser.CondStringContext) { + return new StringConditionValue( + stripQuotes(((LALParser.CondStringContext) ctx).STRING().getText())); + } + if (ctx instanceof LALParser.CondNumberContext) { + return new NumberConditionValue( + Double.parseDouble(((LALParser.CondNumberContext) ctx).NUMBER().getText())); + } + if (ctx instanceof LALParser.CondNullContext) { + return new NullConditionValue(); + } + if (ctx instanceof LALParser.CondValueAccessContext) { + final LALParser.CondValueAccessContext va = + (LALParser.CondValueAccessContext) ctx; + final String cast = va.typeCast() != null ? extractCastType(va.typeCast()) : null; + return new ValueAccessConditionValue(visitValueAccess(va.valueAccess()), cast); + } + // condBool, condFunctionCall + return new StringConditionValue(ctx.getText()); + } + + private static Condition visitConditionExprAsCondition( + final LALParser.ConditionExprContext ctx) { + if (ctx instanceof LALParser.CondValueAccessContext) { + final LALParser.CondValueAccessContext va = + (LALParser.CondValueAccessContext) ctx; + final String cast = va.typeCast() != null ? extractCastType(va.typeCast()) : null; + return new ExprCondition(visitValueAccess(va.valueAccess()), cast); + } + return new ExprCondition( + new ValueAccess(List.of(ctx.getText()), false, false, List.of()), null); + } + + // ==================== Value access ==================== + + private static ValueAccess visitValueAccess(final LALParser.ValueAccessContext ctx) { + final List segments = new ArrayList<>(); + boolean parsedRef = false; + boolean logRef = false; + + final LALParser.ValueAccessPrimaryContext primary = ctx.valueAccessPrimary(); + if (primary instanceof LALParser.ValueParsedContext) { + parsedRef = true; + segments.add("parsed"); + } else if (primary instanceof LALParser.ValueLogContext) { + logRef = true; + segments.add("log"); + } else if (primary instanceof LALParser.ValueIdentifierContext) { + segments.add(((LALParser.ValueIdentifierContext) primary).IDENTIFIER().getText()); + } else if (primary instanceof LALParser.ValueStringContext) { + segments.add(stripQuotes( + ((LALParser.ValueStringContext) primary).STRING().getText())); + } else if (primary instanceof LALParser.ValueNumberContext) { + segments.add(((LALParser.ValueNumberContext) primary).NUMBER().getText()); + } else if (primary instanceof LALParser.ValueFunctionCallContext) { + final LALParser.FunctionInvocationContext fi = + ((LALParser.ValueFunctionCallContext) primary).functionInvocation(); + segments.add(fi.getText()); + } else { + segments.add(primary.getText()); + } + + final List chain = new ArrayList<>(); + for (final LALParser.ValueAccessSegmentContext seg : ctx.valueAccessSegment()) { + if (seg instanceof LALParser.SegmentFieldContext) { + final String name = + ((LALParser.SegmentFieldContext) seg).anyIdentifier().getText(); + segments.add(name); + chain.add(new FieldSegment(name, false)); + } else if (seg instanceof LALParser.SegmentSafeFieldContext) { + final String name = + ((LALParser.SegmentSafeFieldContext) seg).anyIdentifier().getText(); + segments.add(name); + chain.add(new FieldSegment(name, true)); + } else if (seg instanceof LALParser.SegmentMethodContext) { + final LALParser.FunctionInvocationContext fi = + ((LALParser.SegmentMethodContext) seg).functionInvocation(); + segments.add(fi.IDENTIFIER().getText() + "()"); + chain.add(new LALScriptModel.MethodSegment( + fi.IDENTIFIER().getText(), List.of(), false)); + } else if (seg instanceof LALParser.SegmentSafeMethodContext) { + final LALParser.FunctionInvocationContext fi = + ((LALParser.SegmentSafeMethodContext) seg).functionInvocation(); + segments.add(fi.IDENTIFIER().getText() + "()"); + chain.add(new LALScriptModel.MethodSegment( + fi.IDENTIFIER().getText(), List.of(), true)); + } + } + + return new ValueAccess(segments, parsedRef, logRef, chain); + } + + private static String resolveValueAsString(final LALParser.ValueAccessContext ctx) { + final LALParser.ValueAccessPrimaryContext primary = ctx.valueAccessPrimary(); + if (primary instanceof LALParser.ValueStringContext) { + return stripQuotes(((LALParser.ValueStringContext) primary).STRING().getText()); + } + return primary.getText(); + } + + // ==================== Utilities ==================== + + private static String extractCastType(final LALParser.TypeCastContext ctx) { + if (ctx.STRING_TYPE() != null) { + return "String"; + } + if (ctx.LONG_TYPE() != null) { + return "Long"; + } + if (ctx.INTEGER_TYPE() != null) { + return "Integer"; + } + if (ctx.BOOLEAN_TYPE() != null) { + return "Boolean"; + } + return null; + } + + static String stripQuotes(final String s) { + if (s == null || s.length() < 2) { + return s; + } + final char first = s.charAt(0); + if ((first == '\'' || first == '"') && s.charAt(s.length() - 1) == first) { + return s.substring(1, s.length() - 1); + } + // Handle slashy strings: $/ ... /$ + if (s.startsWith("$/") && s.endsWith("/$")) { + return s.substring(2, s.length() - 2); + } + return s; + } + + private static String truncate(final String s, final int maxLen) { + if (s.length() <= maxLen) { + return s; + } + return s.substring(0, maxLen) + "..."; + } +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/BindingAware.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/BindingAware.java new file mode 100644 index 000000000000..628e7f48746d --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/BindingAware.java @@ -0,0 +1,31 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler.rt; + +import org.apache.skywalking.oap.log.analyzer.dsl.Binding; + +/** + * Interface for generated consumer classes that need access to the + * current {@link Binding} at runtime. + */ +public interface BindingAware { + + void setBinding(Binding binding); + + Binding getBinding(); +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/LalExpressionPackageHolder.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/LalExpressionPackageHolder.java new file mode 100644 index 000000000000..815284cf2b3c --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/rt/LalExpressionPackageHolder.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler.rt; + +/** + * Empty marker class used as the class loading anchor for Javassist + * {@code CtClass.toClass(Class)} on JDK 16+. + * Generated LAL expression classes are loaded in this package. + */ +public class LalExpressionPackageHolder { +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java index 763dd64d28ef..ee2beaa47403 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java @@ -13,15 +13,13 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - * */ package org.apache.skywalking.oap.log.analyzer.dsl; +import com.google.protobuf.Descriptors; import com.google.protobuf.Message; -import groovy.lang.Closure; -import groovy.lang.GroovyObjectSupport; -import groovy.lang.MissingPropertyException; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Optional; @@ -31,35 +29,38 @@ import org.apache.skywalking.apm.network.logging.v3.LogData; import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily; import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.DatabaseSlowStatementBuilder; - import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.SampledTraceBuilder; import org.apache.skywalking.oap.server.core.source.Log; /** - * The binding bridge between OAP and the DSL, which provides some convenient methods to ease the use of the raw {@link groovy.lang.Binding#setProperty(java.lang.String, java.lang.Object)} and {@link - * groovy.lang.Binding#getProperty(java.lang.String)}. + * Same-FQCN replacement for upstream Binding. + * Pure Java implementation that does not extend groovy.lang.Binding. + * Uses a simple HashMap for property storage instead of Groovy's binding mechanism. */ -public class Binding extends groovy.lang.Binding { +public class Binding { public static final String KEY_LOG = "log"; - public static final String KEY_PARSED = "parsed"; - public static final String KEY_SAVE = "save"; - public static final String KEY_ABORT = "abort"; - public static final String KEY_METRICS_CONTAINER = "metrics_container"; - public static final String KEY_LOG_CONTAINER = "log_container"; - public static final String KEY_DATABASE_SLOW_STATEMENT = "database_slow_statement"; - public static final String KEY_SAMPLED_TRACE = "sampled_trace"; + private final Map properties = new HashMap<>(); + public Binding() { setProperty(KEY_PARSED, new Parsed()); } + public void setProperty(final String name, final Object value) { + properties.put(name, value); + } + + public Object getProperty(final String name) { + return properties.get(name); + } + public Binding log(final LogData.Builder log) { setProperty(KEY_LOG, log); setProperty(KEY_SAVE, true); @@ -105,7 +106,7 @@ public DatabaseSlowStatementBuilder databaseSlowStatement() { return (DatabaseSlowStatementBuilder) getProperty(KEY_DATABASE_SLOW_STATEMENT); } - public Binding databaseSlowStatement(DatabaseSlowStatementBuilder databaseSlowStatementBuilder) { + public Binding databaseSlowStatement(final DatabaseSlowStatementBuilder databaseSlowStatementBuilder) { setProperty(KEY_DATABASE_SLOW_STATEMENT, databaseSlowStatementBuilder); return this; } @@ -114,7 +115,7 @@ public SampledTraceBuilder sampledTraceBuilder() { return (SampledTraceBuilder) getProperty(KEY_SAMPLED_TRACE); } - public Binding sampledTrace(SampledTraceBuilder sampledTraceBuilder) { + public Binding sampledTrace(final SampledTraceBuilder sampledTraceBuilder) { setProperty(KEY_SAMPLED_TRACE, sampledTraceBuilder); return this; } @@ -142,41 +143,27 @@ public boolean shouldAbort() { return (boolean) getProperty(KEY_ABORT); } - /** - * Set the metrics container to store all metrics generated from the pipeline, - * if no container is set, all generated metrics will be sent to MAL engine for further processing, - * if metrics container is set, all metrics are only stored in the container, and won't be sent to MAL. - * - * @param container the metrics container - */ - public Binding metricsContainer(List container) { + public Binding metricsContainer(final List container) { setProperty(KEY_METRICS_CONTAINER, container); return this; } + @SuppressWarnings("unchecked") public Optional> metricsContainer() { - // noinspection unchecked return Optional.ofNullable((List) getProperty(KEY_METRICS_CONTAINER)); } - /** - * Set the log container to store the final log if it should be persisted in storage, - * if no container is set, the final log will be sent to source receiver, - * if log container is set, the log is only stored in the container, and won't be sent to source receiver. - * - * @param container the log container - */ - public Binding logContainer(AtomicReference container) { + public Binding logContainer(final AtomicReference container) { setProperty(KEY_LOG_CONTAINER, container); return this; } + @SuppressWarnings("unchecked") public Optional> logContainer() { - // noinspection unchecked return Optional.ofNullable((AtomicReference) getProperty(KEY_LOG_CONTAINER)); } - public static class Parsed extends GroovyObjectSupport { + public static class Parsed { @Getter private Matcher matcher; @@ -206,17 +193,20 @@ public Object getAt(final String key) { return null; } - @SuppressWarnings("unused") - public Object propertyMissing(final String name) { - return getAt(name); - } - - static Object getField(Object obj, String name) { - try { - Closure c = new Closure(obj, obj) { - }; - return c.getProperty(name); - } catch (MissingPropertyException ignored) { + static Object getField(final Object obj, final String name) { + if (obj instanceof Message) { + final Descriptors.FieldDescriptor fd = + ((Message) obj).getDescriptorForType().findFieldByName(name); + if (fd != null) { + return ((Message) obj).getField(fd); + } + } + if (obj instanceof Message.Builder) { + final Descriptors.FieldDescriptor fd = + ((Message.Builder) obj).getDescriptorForType().findFieldByName(name); + if (fd != null) { + return ((Message.Builder) obj).getField(fd); + } } return null; } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java index c5666d8061d0..b3f4e4c977d2 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java @@ -13,97 +13,130 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - * */ package org.apache.skywalking.oap.log.analyzer.dsl; -import com.google.common.collect.ImmutableList; - -import groovy.lang.GString; -import groovy.lang.GroovyShell; -import groovy.transform.CompileStatic; -import groovy.util.DelegatingScript; -import java.lang.reflect.Array; -import java.util.List; +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HashMap; import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; import lombok.AccessLevel; import lombok.RequiredArgsConstructor; -import org.apache.skywalking.oap.log.analyzer.dsl.spec.LALDelegatingScript; +import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; -import org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry; import org.apache.skywalking.oap.server.library.module.ModuleManager; import org.apache.skywalking.oap.server.library.module.ModuleStartException; -import org.codehaus.groovy.ast.stmt.DoWhileStatement; -import org.codehaus.groovy.ast.stmt.ForStatement; -import org.codehaus.groovy.ast.stmt.Statement; -import org.codehaus.groovy.ast.stmt.WhileStatement; -import org.codehaus.groovy.control.CompilerConfiguration; -import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer; -import org.codehaus.groovy.control.customizers.ImportCustomizer; -import org.codehaus.groovy.control.customizers.SecureASTCustomizer; - -import static java.util.Collections.singletonList; -import static java.util.Collections.singletonMap; +/** + * Same-FQCN replacement for upstream LAL DSL. + * Loads pre-compiled {@link LalExpression} classes from lal-expressions.txt manifest + * (keyed by SHA-256 hash) instead of Groovy {@code GroovyShell} runtime compilation. + */ +@Slf4j @RequiredArgsConstructor(access = AccessLevel.PRIVATE) public class DSL { - private final DelegatingScript script; + private static final String MANIFEST_PATH = "META-INF/lal-expressions.txt"; + private static volatile Map EXPRESSION_MAP; + private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + private final LalExpression expression; private final FilterSpec filterSpec; + private Binding binding; public static DSL of(final ModuleManager moduleManager, final LogAnalyzerModuleConfig config, final String dsl) throws ModuleStartException { - final CompilerConfiguration cc = new CompilerConfiguration(); - final ASTTransformationCustomizer customizer = - new ASTTransformationCustomizer( - singletonMap( - "extensions", - singletonList(LALPrecompiledExtension.class.getName()) - ), - CompileStatic.class - ); - cc.addCompilationCustomizers(customizer); - final SecureASTCustomizer secureASTCustomizer = new SecureASTCustomizer(); - secureASTCustomizer.setDisallowedStatements( - ImmutableList.>builder() - .add(WhileStatement.class) - .add(DoWhileStatement.class) - .add(ForStatement.class) - .build()); - // noinspection rawtypes - secureASTCustomizer.setAllowedReceiversClasses( - ImmutableList.builder() - .add(Object.class) - .add(Map.class) - .add(List.class) - .add(Array.class) - .add(GString.class) - .add(String.class) - .add(ProcessRegistry.class) - .build()); - cc.addCompilationCustomizers(secureASTCustomizer); - cc.setScriptBaseClass(LALDelegatingScript.class.getName()); - - ImportCustomizer icz = new ImportCustomizer(); - icz.addImport("ProcessRegistry", ProcessRegistry.class.getName()); - cc.addCompilationCustomizers(icz); + final Map exprMap = loadManifest(); + final String dslHash = sha256(dsl); + final String className = exprMap.get(dslHash); + if (className == null) { + throw new ModuleStartException( + "Pre-compiled LAL expression not found for DSL hash: " + dslHash + + ". Available: " + exprMap.size() + " expressions."); + } - final GroovyShell sh = new GroovyShell(cc); - final DelegatingScript script = (DelegatingScript) sh.parse(dsl); - final FilterSpec filterSpec = new FilterSpec(moduleManager, config); - script.setDelegate(filterSpec); - - return new DSL(script, filterSpec); + try { + final Class exprClass = Class.forName(className); + final LalExpression expression = (LalExpression) exprClass.getDeclaredConstructor().newInstance(); + final FilterSpec filterSpec = new FilterSpec(moduleManager, config); + final int count = LOADED_COUNT.incrementAndGet(); + log.debug("Loaded pre-compiled LAL expression [{}/{}]: {}", count, exprMap.size(), className); + return new DSL(expression, filterSpec); + } catch (ClassNotFoundException e) { + throw new ModuleStartException( + "Pre-compiled LAL expression class not found: " + className, e); + } catch (ReflectiveOperationException e) { + throw new ModuleStartException( + "Pre-compiled LAL expression instantiation failed: " + className, e); + } } public void bind(final Binding binding) { + this.binding = binding; this.filterSpec.bind(binding); } public void evaluate() { - script.run(); + expression.execute(filterSpec, binding); + } + + private static Map loadManifest() { + if (EXPRESSION_MAP != null) { + return EXPRESSION_MAP; + } + synchronized (DSL.class) { + if (EXPRESSION_MAP != null) { + return EXPRESSION_MAP; + } + final Map map = new HashMap<>(); + try (InputStream is = DSL.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { + if (is == null) { + log.warn("LAL expression manifest not found: {}", MANIFEST_PATH); + EXPRESSION_MAP = map; + return map; + } + try (BufferedReader reader = new BufferedReader( + new InputStreamReader(is, StandardCharsets.UTF_8))) { + String line; + while ((line = reader.readLine()) != null) { + line = line.trim(); + if (line.isEmpty()) { + continue; + } + final String[] parts = line.split("=", 2); + if (parts.length == 2) { + map.put(parts[0], parts[1]); + } + } + } + } catch (IOException e) { + throw new IllegalStateException("Failed to load LAL expression manifest", e); + } + log.info("Loaded {} pre-compiled LAL expressions from manifest", map.size()); + EXPRESSION_MAP = map; + return map; + } + } + + static String sha256(final String input) { + try { + final MessageDigest digest = MessageDigest.getInstance("SHA-256"); + final byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8)); + final StringBuilder hex = new StringBuilder(); + for (final byte b : hash) { + hex.append(String.format("%02x", b)); + } + return hex.toString(); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException("SHA-256 not available", e); + } } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java index d815095776dc..6b8e947cb831 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java @@ -18,7 +18,7 @@ package org.apache.skywalking.oap.log.analyzer.dsl.spec; -import groovy.lang.Closure; +import java.util.function.Consumer; import lombok.Getter; import lombok.RequiredArgsConstructor; import lombok.experimental.Accessors; @@ -42,15 +42,10 @@ public void bind(final Binding b) { } @SuppressWarnings("unused") - public void abort(final Closure cl) { + public void abort(final Consumer cl) { BINDING.get().abort(); } - @SuppressWarnings("unused") - public Object propertyMissing(final String name) { - return BINDING.get().getVariable(name); - } - @SuppressWarnings("unused") public String tag(String key) { return BINDING.get().log().getTags().getDataList() diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java index 2a51d10d4f1b..02c018670128 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java @@ -20,8 +20,6 @@ import com.google.common.base.Joiner; import com.google.common.collect.ImmutableMap; -import groovy.lang.Closure; -import groovy.lang.DelegatesTo; import java.text.ParseException; import java.text.SimpleDateFormat; import java.util.Collection; @@ -241,99 +239,6 @@ public void layer(final String layer) { } } - @SuppressWarnings("unused") - public void metrics(@DelegatesTo(SampleBuilder.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - final SampleBuilder builder = new SampleBuilder(); - cl.setDelegate(builder); - cl.call(); - - final Sample sample = builder.build(); - final SampleFamily sampleFamily = SampleFamilyBuilder.newBuilder(sample).build(); - - final Optional> possibleMetricsContainer = BINDING.get().metricsContainer(); - - if (possibleMetricsContainer.isPresent()) { - possibleMetricsContainer.get().add(sampleFamily); - } else { - metricConverts.forEach(it -> it.toMeter( - ImmutableMap.builder() - .put(sample.getName(), sampleFamily) - .build() - )); - } - } - - @SuppressWarnings("unused") - public void slowSql(@DelegatesTo(SlowSqlSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - LogData.Builder log = BINDING.get().log(); - if (log.getLayer() == null - || log.getService() == null - || log.getTimestamp() < 1) { - LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); - return; - } - DatabaseSlowStatementBuilder builder = new DatabaseSlowStatementBuilder(namingControl); - builder.setLayer(Layer.nameOf(log.getLayer())); - - builder.setServiceName(log.getService()); - - BINDING.get().databaseSlowStatement(builder); - - cl.setDelegate(slowSql); - cl.call(); - - if (builder.getId() == null - || builder.getLatency() < 1 - || builder.getStatement() == null) { - LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); - return; - } - - long timeBucketForDB = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Second); - builder.setTimeBucket(timeBucketForDB); - builder.setTimestamp(log.getTimestamp()); - - builder.prepare(); - sourceReceiver.receive(builder.toDatabaseSlowStatement()); - - ServiceMeta serviceMeta = new ServiceMeta(); - serviceMeta.setName(builder.getServiceName()); - serviceMeta.setLayer(builder.getLayer()); - long timeBucket = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Minute); - serviceMeta.setTimeBucket(timeBucket); - sourceReceiver.receive(serviceMeta); - } - - @SuppressWarnings("unused") - public void sampledTrace(@DelegatesTo(SampledTraceSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - LogData.Builder log = BINDING.get().log(); - SampledTraceBuilder builder = new SampledTraceBuilder(namingControl); - builder.setLayer(log.getLayer()); - builder.setTimestamp(log.getTimestamp()); - builder.setServiceName(log.getService()); - builder.setServiceInstanceName(log.getServiceInstance()); - builder.setTraceId(log.getTraceContext().getTraceId()); - BINDING.get().sampledTrace(builder); - - cl.setDelegate(sampledTrace); - cl.call(); - - builder.validate(); - final Record record = builder.toRecord(); - final ISource entity = builder.toEntity(); - RecordStreamProcessor.getInstance().in(record); - sourceReceiver.receive(entity); - } - public void metrics(final Consumer consumer) { if (BINDING.get().shouldAbort()) { return; diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java index b83e71b8b849..c9e3338df69a 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java @@ -21,8 +21,6 @@ import com.fasterxml.jackson.core.type.TypeReference; import com.google.protobuf.Message; import com.google.protobuf.TextFormat; -import groovy.lang.Closure; -import groovy.lang.DelegatesTo; import java.util.Arrays; import java.util.List; import java.util.Map; @@ -87,110 +85,6 @@ public FilterSpec(final ModuleManager moduleManager, sink = new SinkSpec(moduleManager(), moduleConfig()); } - @SuppressWarnings("unused") - public void text(@DelegatesTo(TextParserSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - cl.setDelegate(textParser); - cl.call(); - } - - @SuppressWarnings("unused") - public void json(@DelegatesTo(JsonParserSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - cl.setDelegate(jsonParser); - cl.call(); - - final LogData.Builder logData = BINDING.get().log(); - try { - - final Map parsed = jsonParser.create().readValue( - logData.getBody().getJson().getJson(), parsedType - ); - - BINDING.get().parsed(parsed); - } catch (final Exception e) { - if (jsonParser.abortOnFailure()) { - BINDING.get().abort(); - } - } - } - - @SuppressWarnings({"unused"}) - public void yaml(@DelegatesTo(YamlParserSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - cl.setDelegate(yamlParser); - cl.call(); - - final LogData.Builder logData = BINDING.get().log(); - try { - final Map parsed = yamlParser.create().load( - logData.getBody().getYaml().getYaml() - ); - - BINDING.get().parsed(parsed); - } catch (final Exception e) { - if (yamlParser.abortOnFailure()) { - BINDING.get().abort(); - } - } - } - - @SuppressWarnings("unused") - public void extractor(@DelegatesTo(ExtractorSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - cl.setDelegate(extractor); - cl.call(); - } - - @SuppressWarnings("unused") - public void sink(@DelegatesTo(SinkSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - cl.setDelegate(sink); - cl.call(); - - final Binding b = BINDING.get(); - final LogData.Builder logData = b.log(); - final Message extraLog = b.extraLog(); - - if (!b.shouldSave()) { - if (LOGGER.isDebugEnabled()) { - LOGGER.debug("Log is dropped: {}", TextFormat.shortDebugString(logData)); - } - return; - } - - final Optional> container = BINDING.get().logContainer(); - if (container.isPresent()) { - sinkListenerFactories.stream() - .map(LogSinkListenerFactory::create) - .filter(it -> it instanceof RecordSinkListener) - .map(it -> it.parse(logData, extraLog)) - .map(it -> (RecordSinkListener) it) - .map(RecordSinkListener::getLog) - .findFirst() - .ifPresent(log -> container.get().set(log)); - } else { - sinkListenerFactories.stream() - .map(LogSinkListenerFactory::create) - .forEach(it -> it.parse(logData, extraLog).build()); - } - } - - @SuppressWarnings("unused") - public void filter(final Closure cl) { - cl.call(); - } - public void text(final Consumer consumer) { if (BINDING.get().shouldAbort()) { return; diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java index f10d471ee184..7e1ff921adfe 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java @@ -18,21 +18,16 @@ package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink; -import groovy.lang.Closure; -import groovy.lang.DelegatesTo; -import groovy.lang.GString; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Consumer; import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; -import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.PossibilitySampler; import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.RateLimitingSampler; import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.Sampler; import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; import org.apache.skywalking.oap.server.library.module.ModuleManager; public class SamplerSpec extends AbstractSpec { - private final Map rateLimitSamplers; private final Map rateLimitSamplersByString; private final Map possibilitySamplers; private final RateLimitingSampler.ResetHandler rlsResetHandler; @@ -41,26 +36,11 @@ public SamplerSpec(final ModuleManager moduleManager, final LogAnalyzerModuleConfig moduleConfig) { super(moduleManager, moduleConfig); - rateLimitSamplers = new ConcurrentHashMap<>(); rateLimitSamplersByString = new ConcurrentHashMap<>(); possibilitySamplers = new ConcurrentHashMap<>(); rlsResetHandler = new RateLimitingSampler.ResetHandler(); } - @SuppressWarnings("unused") - public void rateLimit(final GString id, @DelegatesTo(RateLimitingSampler.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - - final Sampler sampler = rateLimitSamplers.computeIfAbsent(id, $ -> new RateLimitingSampler(rlsResetHandler).start()); - - cl.setDelegate(sampler); - cl.call(); - - sampleWith(sampler); - } - @SuppressWarnings("unused") public void rateLimit(final String id, final Consumer consumer) { if (BINDING.get().shouldAbort()) { @@ -75,20 +55,6 @@ public void rateLimit(final String id, final Consumer consu sampleWith(sampler); } - @SuppressWarnings("unused") - public void possibility(final int percentage, @DelegatesTo(PossibilitySampler.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - - final Sampler sampler = possibilitySamplers.computeIfAbsent(percentage, $ -> new PossibilitySampler(percentage).start()); - - cl.setDelegate(sampler); - cl.call(); - - sampleWith(sampler); - } - private void sampleWith(final Sampler sampler) { if (BINDING.get().shouldAbort()) { return; diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java index f2ae371a21e2..631cccfca93c 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java @@ -18,8 +18,6 @@ package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink; -import groovy.lang.Closure; -import groovy.lang.DelegatesTo; import java.util.function.Consumer; import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; @@ -36,15 +34,6 @@ public SinkSpec(final ModuleManager moduleManager, sampler = new SamplerSpec(moduleManager(), moduleConfig()); } - @SuppressWarnings("unused") - public void sampler(@DelegatesTo(SamplerSpec.class) final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - cl.setDelegate(sampler); - cl.call(); - } - public void sampler(final Consumer consumer) { if (BINDING.get().shouldAbort()) { return; @@ -52,14 +41,6 @@ public void sampler(final Consumer consumer) { consumer.accept(sampler); } - @SuppressWarnings("unused") - public void enforcer(final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - BINDING.get().save(); - } - public void enforcer() { if (BINDING.get().shouldAbort()) { return; @@ -67,14 +48,6 @@ public void enforcer() { BINDING.get().save(); } - @SuppressWarnings("unused") - public void dropper(final Closure cl) { - if (BINDING.get().shouldAbort()) { - return; - } - BINDING.get().drop(); - } - public void dropper() { if (BINDING.get().shouldAbort()) { return; diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java new file mode 100644 index 000000000000..7bde461a4f76 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java @@ -0,0 +1,97 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler; + +import javassist.ClassPool; +import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertNotNull; + +class LALClassGeneratorTest { + + private LALClassGenerator generator; + + @BeforeEach + void setUp() { + generator = new LALClassGenerator(new ClassPool(true)); + } + + @Test + void compileMinimalFilter() throws Exception { + final LalExpression expr = generator.compile( + "filter { sink {} }"); + assertNotNull(expr); + } + + @Test + void compileJsonParserFilter() throws Exception { + final LalExpression expr = generator.compile( + "filter { json {} sink {} }"); + assertNotNull(expr); + } + + @Test + void compileJsonWithExtractor() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed.service as String\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileTextWithRegexp() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " text {\n" + + " regexp '(?\\\\d+) (?\\\\w+) (?.*)'\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileSinkWithEnforcer() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " sink {\n" + + " enforcer {}\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void generateSourceReturnsJavaCode() { + final String source = generator.generateSource( + "filter { json {} sink {} }"); + assertNotNull(source); + org.junit.jupiter.api.Assertions.assertTrue( + source.contains("filterSpec.json()")); + org.junit.jupiter.api.Assertions.assertTrue( + source.contains("filterSpec.sink()")); + } +} diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java new file mode 100644 index 000000000000..5c29397f8c4c --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java @@ -0,0 +1,192 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.compiler; + +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class LALScriptParserTest { + + @Test + void parseMinimalFilter() { + final LALScriptModel model = LALScriptParser.parse("filter { sink {} }"); + assertNotNull(model); + assertEquals(1, model.getStatements().size()); + assertInstanceOf(LALScriptModel.SinkBlock.class, model.getStatements().get(0)); + } + + @Test + void parseJsonParserWithExtractorAndSink() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed.service as String\n" + + " layer parsed.layer as String\n" + + " }\n" + + " sink {}\n" + + "}"); + + assertEquals(3, model.getStatements().size()); + assertInstanceOf(LALScriptModel.JsonParser.class, model.getStatements().get(0)); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(1); + assertEquals(2, extractor.getStatements().size()); + + final LALScriptModel.FieldAssignment serviceField = + (LALScriptModel.FieldAssignment) extractor.getStatements().get(0); + assertEquals(LALScriptModel.FieldType.SERVICE, serviceField.getFieldType()); + assertTrue(serviceField.getValue().isParsedRef()); + assertEquals("String", serviceField.getCastType()); + } + + @Test + void parseTextParserWithRegexp() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " text {\n" + + " regexp '.+\"(?.+)\"(?\\d{3}).+'\n" + + " }\n" + + " sink {}\n" + + "}"); + + assertEquals(2, model.getStatements().size()); + final LALScriptModel.TextParser textParser = + (LALScriptModel.TextParser) model.getStatements().get(0); + assertNotNull(textParser.getRegexpPattern()); + } + + @Test + void parseSlowSql() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " layer parsed.layer as String\n" + + " service parsed.service as String\n" + + " timestamp parsed.time as String\n" + + " slowSql {\n" + + " id parsed.id as String\n" + + " statement parsed.statement as String\n" + + " latency parsed.query_time as Long\n" + + " }\n" + + " }\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(1); + + // Find the slowSql block + LALScriptModel.SlowSqlBlock slowSql = null; + for (final LALScriptModel.ExtractorStatement stmt : extractor.getStatements()) { + if (stmt instanceof LALScriptModel.SlowSqlBlock) { + slowSql = (LALScriptModel.SlowSqlBlock) stmt; + } + } + assertNotNull(slowSql); + assertNotNull(slowSql.getId()); + assertEquals("String", slowSql.getIdCast()); + assertNotNull(slowSql.getStatement()); + assertEquals("String", slowSql.getStatementCast()); + assertNotNull(slowSql.getLatency()); + assertEquals("Long", slowSql.getLatencyCast()); + } + + @Test + void parseMetricsBlock() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " extractor {\n" + + " metrics {\n" + + " timestamp log.timestamp as Long\n" + + " labels level: parsed.level, service: log.service\n" + + " name \"nginx_error_log_count\"\n" + + " value 1\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(0); + final LALScriptModel.MetricsBlock metrics = + (LALScriptModel.MetricsBlock) extractor.getStatements().get(0); + + assertEquals("nginx_error_log_count", metrics.getName()); + assertEquals(2, metrics.getLabels().size()); + assertTrue(metrics.getLabels().containsKey("level")); + assertTrue(metrics.getLabels().containsKey("service")); + assertNotNull(metrics.getValue()); + } + + @Test + void parseSinkWithSampler() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit('service:error') {\n" + + " rpm 6000\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + + final LALScriptModel.SinkBlock sink = + (LALScriptModel.SinkBlock) model.getStatements().get(0); + assertEquals(1, sink.getStatements().size()); + final LALScriptModel.SamplerBlock sampler = + (LALScriptModel.SamplerBlock) sink.getStatements().get(0); + assertEquals(1, sampler.getContents().size()); + final LALScriptModel.RateLimitBlock rateLimit = + (LALScriptModel.RateLimitBlock) sampler.getContents().get(0); + assertEquals("service:error", rateLimit.getId()); + assertEquals(6000, rateLimit.getRpm()); + } + + @Test + void parseIfCondition() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " if (parsed.status) {\n" + + " extractor {\n" + + " layer parsed.layer as String\n" + + " }\n" + + " sink {}\n" + + " }\n" + + "}"); + + assertEquals(1, model.getStatements().size()); + final LALScriptModel.IfBlock ifBlock = + (LALScriptModel.IfBlock) model.getStatements().get(0); + assertNotNull(ifBlock.getCondition()); + assertEquals(2, ifBlock.getThenBranch().size()); + } + + @Test + void parseSyntaxErrorThrows() { + assertThrows(IllegalArgumentException.class, + () -> LALScriptParser.parse("filter {")); + } +} diff --git a/oap-server/analyzer/log-analyzer-v2/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java similarity index 100% rename from oap-server/analyzer/log-analyzer-v2/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java rename to oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java diff --git a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java b/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java deleted file mode 100644 index c1deeb2a04c6..000000000000 --- a/oap-server/analyzer/mal-transpiler/src/main/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspiler.java +++ /dev/null @@ -1,1099 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.server.transpiler.mal; - -import java.io.File; -import java.io.IOException; -import java.io.StringWriter; -import java.nio.file.Files; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.LinkedHashMap; -import java.util.LinkedHashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.stream.Collectors; -import javax.tools.JavaCompiler; -import javax.tools.JavaFileObject; -import javax.tools.StandardJavaFileManager; -import javax.tools.ToolProvider; -import lombok.extern.slf4j.Slf4j; -import org.codehaus.groovy.ast.ClassNode; -import org.codehaus.groovy.ast.ModuleNode; -import org.codehaus.groovy.ast.Parameter; -import org.codehaus.groovy.ast.expr.ArgumentListExpression; -import org.codehaus.groovy.ast.expr.BinaryExpression; -import org.codehaus.groovy.ast.expr.ClosureExpression; -import org.codehaus.groovy.ast.expr.ClassExpression; -import org.codehaus.groovy.ast.expr.ConstantExpression; -import org.codehaus.groovy.ast.expr.DeclarationExpression; -import org.codehaus.groovy.ast.expr.ElvisOperatorExpression; -import org.codehaus.groovy.ast.expr.Expression; -import org.codehaus.groovy.ast.expr.ListExpression; -import org.codehaus.groovy.ast.expr.MapEntryExpression; -import org.codehaus.groovy.ast.expr.MapExpression; -import org.codehaus.groovy.ast.expr.MethodCallExpression; -import org.codehaus.groovy.ast.expr.PropertyExpression; -import org.codehaus.groovy.ast.expr.TernaryExpression; -import org.codehaus.groovy.ast.expr.TupleExpression; -import org.codehaus.groovy.ast.expr.VariableExpression; -import org.codehaus.groovy.syntax.Types; -import org.codehaus.groovy.ast.expr.BooleanExpression; -import org.codehaus.groovy.ast.expr.NotExpression; -import org.codehaus.groovy.ast.stmt.BlockStatement; -import org.codehaus.groovy.ast.stmt.EmptyStatement; -import org.codehaus.groovy.ast.stmt.ExpressionStatement; -import org.codehaus.groovy.ast.stmt.IfStatement; -import org.codehaus.groovy.ast.stmt.ReturnStatement; -import org.codehaus.groovy.ast.stmt.Statement; -import org.codehaus.groovy.control.CompilationUnit; -import org.codehaus.groovy.control.CompilerConfiguration; -import org.codehaus.groovy.control.Phases; - -/** - * Transpiles Groovy MAL expressions to Java source code at build time. - * Parses expression strings into Groovy AST (via CompilationUnit at CONVERSION phase), - * walks AST nodes, and produces equivalent Java classes implementing MalExpression or MalFilter. - * - *

Supported AST patterns: - *

    - *
  • Variable references: sample family lookups, DownsamplingType constants, KNOWN_TYPES
    • - *
  • Method chains: .sum(), .service(), .tagEqual(), .rate(), .histogram(), etc.
    • - *
  • Binary arithmetic with operand-swap logic per upstream ExpandoMetaClass - * (N-SF: sf.minus(N).negative(), N/SF: sf.newValue(v->N/v))
    • - *
  • tag() closures: TagFunction lambda (assignment, remove, string concat, if/else)
    • - *
  • Filter closures: MalFilter class (==, !=, in, truthiness, negation, &&, ||)
    • - *
  • forEach() closures: ForEachFunction lambda (var decls, if/else-if, early return)
    • - *
  • instance() with PropertiesExtractor closure: Map.of() from map literals
    • - *
  • Elvis (?:), safe navigation (?.), ternary (? :)
    • - *
  • Batch compilation via javax.tools.JavaCompiler + manifest writing
    • - *
- */ -@Slf4j -public class MalToJavaTranspiler { - - public static final String GENERATED_PACKAGE = - "org.apache.skywalking.oap.server.core.source.oal.rt.mal"; - - private static final Set DOWNSAMPLING_CONSTANTS = Set.of( - "AVG", "SUM", "LATEST", "SUM_PER_MIN", "MAX", "MIN" - ); - - private static final Set KNOWN_TYPES = Set.of( - "Layer", "DetectPoint", "K8sRetagType", "ProcessRegistry", "TimeUnit" - ); - - // ---- Batch state tracking ---- - - private final Map expressionSources = new LinkedHashMap<>(); - - private final Map filterSources = new LinkedHashMap<>(); - - private final Map filterLiteralToClass = new LinkedHashMap<>(); - - /** - * Transpile a MAL expression to a Java class source implementing MalExpression. - * - * @param className simple class name (e.g. "MalExpr_meter_jvm_heap") - * @param expression the Groovy expression string - * @return generated Java source code - */ - public String transpileExpression(final String className, final String expression) { - final ModuleNode ast = parseToAST(expression); - final Statement body = extractBody(ast); - - final Set sampleNames = new LinkedHashSet<>(); - collectSampleNames(body, sampleNames); - - final String javaBody = visitStatement(body); - - final StringBuilder sb = new StringBuilder(); - sb.append("package ").append(GENERATED_PACKAGE).append(";\n\n"); - sb.append("import java.util.*;\n"); - sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.*;\n"); - sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.*;\n"); - sb.append("import org.apache.skywalking.oap.server.core.analysis.Layer;\n"); - sb.append("import org.apache.skywalking.oap.server.core.source.DetectPoint;\n"); - sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType;\n"); - sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry;\n\n"); - - sb.append("public class ").append(className).append(" implements MalExpression {\n"); - sb.append(" @Override\n"); - sb.append(" public SampleFamily run(Map samples) {\n"); - - if (!sampleNames.isEmpty()) { - sb.append(" ExpressionParsingContext.get().ifPresent(ctx -> {\n"); - for (String name : sampleNames) { - sb.append(" ctx.getSamples().add(\"").append(escapeJava(name)).append("\");\n"); - } - sb.append(" });\n"); - } - - sb.append(" return ").append(javaBody).append(";\n"); - sb.append(" }\n"); - sb.append("}\n"); - - return sb.toString(); - } - - /** - * Transpile a MAL filter literal to a Java class source implementing MalFilter. - * Filter literals are closures like: { tags -> tags.job_name == 'vm-monitoring' } - * - * @param className simple class name (e.g. "MalFilter_0") - * @param filterLiteral the Groovy closure literal string - * @return generated Java source code - */ - public String transpileFilter(final String className, final String filterLiteral) { - final ModuleNode ast = parseToAST(filterLiteral); - final Statement body = extractBody(ast); - - // The filter literal is a closure expression at the top level - final ClosureExpression closure = extractClosure(body); - final Parameter[] params = closure.getParameters(); - final String tagsVar = (params != null && params.length > 0) ? params[0].getName() : "tags"; - - // Get the body expression — may need to unwrap inner block/closure - final Expression bodyExpr = extractFilterBodyExpr(closure.getCode(), tagsVar); - - // Generate the boolean condition - final String condition = visitFilterCondition(bodyExpr, tagsVar); - - final StringBuilder sb = new StringBuilder(); - sb.append("package ").append(GENERATED_PACKAGE).append(";\n\n"); - sb.append("import java.util.*;\n"); - sb.append("import org.apache.skywalking.oap.meter.analyzer.dsl.*;\n\n"); - - sb.append("public class ").append(className).append(" implements MalFilter {\n"); - sb.append(" @Override\n"); - sb.append(" public boolean test(Map tags) {\n"); - sb.append(" return ").append(condition).append(";\n"); - sb.append(" }\n"); - sb.append("}\n"); - - return sb.toString(); - } - - private ClosureExpression extractClosure(final Statement body) { - final List stmts = getStatements(body); - if (stmts.size() == 1 && stmts.get(0) instanceof ExpressionStatement) { - final Expression expr = ((ExpressionStatement) stmts.get(0)).getExpression(); - if (expr instanceof ClosureExpression) { - return (ClosureExpression) expr; - } - } - throw new IllegalStateException( - "Filter literal must be a single closure expression, got: " - + (stmts.isEmpty() ? "empty" : stmts.get(0).getClass().getSimpleName())); - } - - private Expression extractFilterBodyExpr(final Statement code, final String tagsVar) { - final List stmts = getStatements(code); - if (stmts.isEmpty()) { - throw new IllegalStateException("Empty filter closure body"); - } - - final Statement last = stmts.get(stmts.size() - 1); - Expression expr; - if (last instanceof ExpressionStatement) { - expr = ((ExpressionStatement) last).getExpression(); - } else if (last instanceof ReturnStatement) { - expr = ((ReturnStatement) last).getExpression(); - } else if (last instanceof BlockStatement) { - return extractFilterBodyExpr(last, tagsVar); - } else { - throw new UnsupportedOperationException( - "Unsupported filter body statement: " + last.getClass().getSimpleName()); - } - - if (expr instanceof ClosureExpression) { - final ClosureExpression inner = (ClosureExpression) expr; - return extractFilterBodyExpr(inner.getCode(), tagsVar); - } - - return expr; - } - - // ---- AST Parsing ---- - - ModuleNode parseToAST(final String expression) { - final CompilerConfiguration cc = new CompilerConfiguration(); - final CompilationUnit cu = new CompilationUnit(cc); - cu.addSource("Script", expression); - cu.compile(Phases.CONVERSION); - final List modules = cu.getAST().getModules(); - if (modules.isEmpty()) { - throw new IllegalStateException("No AST modules produced for: " + expression); - } - return modules.get(0); - } - - Statement extractBody(final ModuleNode module) { - final BlockStatement block = module.getStatementBlock(); - if (block != null && !block.getStatements().isEmpty()) { - return block; - } - final List classes = module.getClasses(); - if (!classes.isEmpty()) { - return module.getStatementBlock(); - } - throw new IllegalStateException("Empty AST body"); - } - - // ---- Sample Name Collection ---- - - private void collectSampleNames(final Statement stmt, final Set names) { - if (stmt instanceof BlockStatement) { - for (Statement s : ((BlockStatement) stmt).getStatements()) { - collectSampleNames(s, names); - } - } else if (stmt instanceof ExpressionStatement) { - collectSampleNamesFromExpr(((ExpressionStatement) stmt).getExpression(), names); - } else if (stmt instanceof ReturnStatement) { - collectSampleNamesFromExpr(((ReturnStatement) stmt).getExpression(), names); - } - } - - void collectSampleNamesFromExpr(final Expression expr, final Set names) { - if (expr instanceof VariableExpression) { - final String name = ((VariableExpression) expr).getName(); - if (!DOWNSAMPLING_CONSTANTS.contains(name) - && !KNOWN_TYPES.contains(name) - && !name.equals("this") && !name.equals("time")) { - names.add(name); - } - } else if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - collectSampleNamesFromExpr(bin.getLeftExpression(), names); - collectSampleNamesFromExpr(bin.getRightExpression(), names); - } else if (expr instanceof PropertyExpression) { - final PropertyExpression pe = (PropertyExpression) expr; - collectSampleNamesFromExpr(pe.getObjectExpression(), names); - } else if (expr instanceof MethodCallExpression) { - final MethodCallExpression mce = (MethodCallExpression) expr; - collectSampleNamesFromExpr(mce.getObjectExpression(), names); - collectSampleNamesFromExpr(mce.getArguments(), names); - } else if (expr instanceof ArgumentListExpression) { - for (Expression e : ((ArgumentListExpression) expr).getExpressions()) { - collectSampleNamesFromExpr(e, names); - } - } else if (expr instanceof TupleExpression) { - for (Expression e : ((TupleExpression) expr).getExpressions()) { - collectSampleNamesFromExpr(e, names); - } - } - } - - // ---- Statement Visiting ---- - - String visitStatement(final Statement stmt) { - if (stmt instanceof BlockStatement) { - final List stmts = ((BlockStatement) stmt).getStatements(); - if (stmts.size() == 1) { - return visitStatement(stmts.get(0)); - } - // Multi-statement: last one is the return value - return visitStatement(stmts.get(stmts.size() - 1)); - } else if (stmt instanceof ExpressionStatement) { - return visitExpression(((ExpressionStatement) stmt).getExpression()); - } else if (stmt instanceof ReturnStatement) { - return visitExpression(((ReturnStatement) stmt).getExpression()); - } - throw new UnsupportedOperationException( - "Unsupported statement: " + stmt.getClass().getSimpleName()); - } - - // ---- Expression Visiting ---- - - String visitExpression(final Expression expr) { - if (expr instanceof VariableExpression) { - return visitVariable((VariableExpression) expr); - } else if (expr instanceof ConstantExpression) { - return visitConstant((ConstantExpression) expr); - } else if (expr instanceof MethodCallExpression) { - return visitMethodCall((MethodCallExpression) expr); - } else if (expr instanceof PropertyExpression) { - return visitProperty((PropertyExpression) expr); - } else if (expr instanceof ListExpression) { - return visitList((ListExpression) expr); - } else if (expr instanceof BinaryExpression) { - return visitBinary((BinaryExpression) expr); - } else if (expr instanceof ClosureExpression) { - throw new UnsupportedOperationException( - "Bare ClosureExpression outside method call context: " + expr.getText()); - } - throw new UnsupportedOperationException( - "Unsupported expression (not yet implemented): " - + expr.getClass().getSimpleName() + " = " + expr.getText()); - } - - private String visitVariable(final VariableExpression expr) { - final String name = expr.getName(); - if (DOWNSAMPLING_CONSTANTS.contains(name)) { - return "DownsamplingType." + name; - } - if (KNOWN_TYPES.contains(name)) { - return name; - } - if (name.equals("this")) { - return "this"; - } - // Sample family lookup - return "samples.getOrDefault(\"" + escapeJava(name) + "\", SampleFamily.EMPTY)"; - } - - private String visitConstant(final ConstantExpression expr) { - final Object value = expr.getValue(); - if (value == null) { - return "null"; - } - if (value instanceof String) { - return "\"" + escapeJava((String) value) + "\""; - } - if (value instanceof Integer) { - return value.toString(); - } - if (value instanceof Long) { - return value + "L"; - } - if (value instanceof Double) { - return value.toString(); - } - if (value instanceof Float) { - return value + "f"; - } - if (value instanceof Boolean) { - return value.toString(); - } - return value.toString(); - } - - // ---- MethodCall, Property, List ---- - - private String visitMethodCall(final MethodCallExpression expr) { - final String methodName = expr.getMethodAsString(); - final Expression objExpr = expr.getObjectExpression(); - final ArgumentListExpression args = toArgList(expr.getArguments()); - - // tag(closure) -> TagFunction lambda - if ("tag".equals(methodName) && args.getExpressions().size() == 1 - && args.getExpression(0) instanceof ClosureExpression) { - final String obj = visitExpression(objExpr); - final String lambda = visitTagClosure((ClosureExpression) args.getExpression(0)); - return obj + ".tag((TagFunction) " + lambda + ")"; - } - - // forEach(list, closure) -> ForEachFunction lambda - if ("forEach".equals(methodName) && args.getExpressions().size() == 2 - && args.getExpression(1) instanceof ClosureExpression) { - final String obj = visitExpression(objExpr); - final String list = visitExpression(args.getExpression(0)); - final String lambda = visitForEachClosure((ClosureExpression) args.getExpression(1)); - return obj + ".forEach(" + list + ", (ForEachFunction) " + lambda + ")"; - } - - // instance(..., closure) -> last arg is PropertiesExtractor lambda - if ("instance".equals(methodName) && !args.getExpressions().isEmpty()) { - final Expression lastArg = args.getExpression(args.getExpressions().size() - 1); - if (lastArg instanceof ClosureExpression) { - final String obj = visitExpression(objExpr); - final List argStrs = new ArrayList<>(); - for (int i = 0; i < args.getExpressions().size() - 1; i++) { - argStrs.add(visitExpression(args.getExpression(i))); - } - final String lambda = visitPropertiesExtractorClosure((ClosureExpression) lastArg); - argStrs.add("(PropertiesExtractor) " + lambda); - return obj + ".instance(" + String.join(", ", argStrs) + ")"; - } - } - - final String obj = visitExpression(objExpr); - - // Static method calls: ClassExpression.method(...) - if (objExpr instanceof ClassExpression) { - final String typeName = objExpr.getType().getNameWithoutPackage(); - final List argStrs = visitArgList(args); - return typeName + "." + methodName + "(" + String.join(", ", argStrs) + ")"; - } - - // Regular instance method call: obj.method(args) - final List argStrs = visitArgList(args); - return obj + "." + methodName + "(" + String.join(", ", argStrs) + ")"; - } - - private String visitProperty(final PropertyExpression expr) { - final Expression obj = expr.getObjectExpression(); - final String prop = expr.getPropertyAsString(); - - if (obj instanceof ClassExpression) { - return obj.getType().getNameWithoutPackage() + "." + prop; - } - if (obj instanceof VariableExpression) { - final String varName = ((VariableExpression) obj).getName(); - if (KNOWN_TYPES.contains(varName)) { - return varName + "." + prop; - } - } - - return visitExpression(obj) + "." + prop; - } - - private String visitList(final ListExpression expr) { - final List elements = new ArrayList<>(); - for (Expression e : expr.getExpressions()) { - elements.add(visitExpression(e)); - } - return "List.of(" + String.join(", ", elements) + ")"; - } - - // ---- Binary Arithmetic ---- - - private String visitBinary(final BinaryExpression expr) { - final int opType = expr.getOperation().getType(); - - if (isArithmetic(opType)) { - return visitArithmetic(expr.getLeftExpression(), expr.getRightExpression(), opType); - } - - throw new UnsupportedOperationException( - "Unsupported binary operator (not yet implemented): " - + expr.getOperation().getText() + " in " + expr.getText()); - } - - /** - * Arithmetic with operand-swap logic per upstream ExpandoMetaClass: - * 
-     *   SF + SF  -> left.plus(right)
-     *   SF - SF  -> left.minus(right)
-     *   SF * SF  -> left.multiply(right)
-     *   SF / SF  -> left.div(right)
-     *   SF op N  -> sf.op(N)
-     *   N + SF   -> sf.plus(N)          (swap)
-     *   N - SF   -> sf.minus(N).negative()
-     *   N * SF   -> sf.multiply(N)      (swap)
-     *   N / SF   -> sf.newValue(v -> N / v)
-     *   N op N   -> plain arithmetic
-     *
- */ - private String visitArithmetic(final Expression left, final Expression right, final int opType) { - final boolean leftNum = isNumberLiteral(left); - final boolean rightNum = isNumberLiteral(right); - final String leftStr = visitExpression(left); - final String rightStr = visitExpression(right); - - if (leftNum && rightNum) { - return "(" + leftStr + " " + opSymbol(opType) + " " + rightStr + ")"; - } - - if (!leftNum && rightNum) { - return leftStr + "." + opMethod(opType) + "(" + rightStr + ")"; - } - - if (leftNum && !rightNum) { - switch (opType) { - case Types.PLUS: - return rightStr + ".plus(" + leftStr + ")"; - case Types.MINUS: - return rightStr + ".minus(" + leftStr + ").negative()"; - case Types.MULTIPLY: - return rightStr + ".multiply(" + leftStr + ")"; - case Types.DIVIDE: - return rightStr + ".newValue(v -> " + leftStr + " / v)"; - default: - break; - } - } - - // SF op SF - return leftStr + "." + opMethod(opType) + "(" + rightStr + ")"; - } - - private boolean isNumberLiteral(final Expression expr) { - if (expr instanceof ConstantExpression) { - return ((ConstantExpression) expr).getValue() instanceof Number; - } - return false; - } - - private boolean isArithmetic(final int opType) { - return opType == Types.PLUS || opType == Types.MINUS - || opType == Types.MULTIPLY || opType == Types.DIVIDE; - } - - private String opMethod(final int opType) { - switch (opType) { - case Types.PLUS: return "plus"; - case Types.MINUS: return "minus"; - case Types.MULTIPLY: return "multiply"; - case Types.DIVIDE: return "div"; - default: return "???"; - } - } - - private String opSymbol(final int opType) { - switch (opType) { - case Types.PLUS: return "+"; - case Types.MINUS: return "-"; - case Types.MULTIPLY: return "*"; - case Types.DIVIDE: return "/"; - default: return "?"; - } - } - - // ---- tag() Closure ---- - - private String visitTagClosure(final ClosureExpression closure) { - final Parameter[] params = closure.getParameters(); - final String tagsVar = (params != null && params.length > 0) ? params[0].getName() : "tags"; - final List stmts = getStatements(closure.getCode()); - - final StringBuilder sb = new StringBuilder(); - sb.append("(").append(tagsVar).append(" -> {\n"); - for (Statement s : stmts) { - sb.append(" ").append(visitTagStatement(s, tagsVar)).append("\n"); - } - sb.append(" return ").append(tagsVar).append(";\n"); - sb.append(" })"); - return sb.toString(); - } - - private String visitTagStatement(final Statement stmt, final String tagsVar) { - if (stmt instanceof ExpressionStatement) { - return visitTagExpr(((ExpressionStatement) stmt).getExpression(), tagsVar) + ";"; - } - if (stmt instanceof ReturnStatement) { - return "return " + tagsVar + ";"; - } - if (stmt instanceof IfStatement) { - return visitTagIf((IfStatement) stmt, tagsVar); - } - throw new UnsupportedOperationException( - "Unsupported tag closure statement: " + stmt.getClass().getSimpleName()); - } - - // ---- If/Else + Compound Conditions in tag() ---- - - private String visitTagIf(final IfStatement ifStmt, final String tagsVar) { - final String condition = visitTagCondition(ifStmt.getBooleanExpression().getExpression(), tagsVar); - final List ifBody = getStatements(ifStmt.getIfBlock()); - final Statement elseBlock = ifStmt.getElseBlock(); - - final StringBuilder sb = new StringBuilder(); - sb.append("if (").append(condition).append(") {\n"); - for (Statement s : ifBody) { - sb.append(" ").append(visitTagStatement(s, tagsVar)).append("\n"); - } - sb.append(" }"); - - if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { - sb.append(" else {\n"); - final List elseBody = getStatements(elseBlock); - for (Statement s : elseBody) { - sb.append(" ").append(visitTagStatement(s, tagsVar)).append("\n"); - } - sb.append(" }"); - } - - return sb.toString(); - } - - private String visitTagCondition(final Expression expr, final String tagsVar) { - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - final int opType = bin.getOperation().getType(); - - if (opType == Types.COMPARE_EQUAL) { - return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, false); - } - if (opType == Types.COMPARE_NOT_EQUAL) { - return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, true); - } - if (opType == Types.LOGICAL_OR) { - return visitTagCondition(bin.getLeftExpression(), tagsVar) - + " || " + visitTagCondition(bin.getRightExpression(), tagsVar); - } - if (opType == Types.LOGICAL_AND) { - return visitTagCondition(bin.getLeftExpression(), tagsVar) - + " && " + visitTagCondition(bin.getRightExpression(), tagsVar); - } - } - if (expr instanceof BooleanExpression) { - return visitTagCondition(((BooleanExpression) expr).getExpression(), tagsVar); - } - return visitTagValue(expr, tagsVar); - } - - private String visitTagEquals(final Expression left, final Expression right, - final String tagsVar, final boolean negate) { - if (isNullConstant(right)) { - final String leftStr = visitTagValue(left, tagsVar); - return negate ? leftStr + " != null" : leftStr + " == null"; - } - if (isNullConstant(left)) { - final String rightStr = visitTagValue(right, tagsVar); - return negate ? rightStr + " != null" : rightStr + " == null"; - } - - final String leftStr = visitTagValue(left, tagsVar); - final String rightStr = visitTagValue(right, tagsVar); - - if (right instanceof ConstantExpression && ((ConstantExpression) right).getValue() instanceof String) { - final String result = rightStr + ".equals(" + leftStr + ")"; - return negate ? "!" + result : result; - } - if (left instanceof ConstantExpression && ((ConstantExpression) left).getValue() instanceof String) { - final String result = leftStr + ".equals(" + rightStr + ")"; - return negate ? "!" + result : result; - } - final String result = "Objects.equals(" + leftStr + ", " + rightStr + ")"; - return negate ? "!" + result : result; - } - - // ---- Filter Conditions ---- - - private String visitFilterCondition(final Expression expr, final String tagsVar) { - if (expr instanceof NotExpression) { - final Expression inner = ((NotExpression) expr).getExpression(); - final String val = visitTagValue(inner, tagsVar); - return "(" + val + " == null || " + val + ".isEmpty())"; - } - - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - final int opType = bin.getOperation().getType(); - - if (opType == Types.COMPARE_EQUAL) { - return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, false); - } - if (opType == Types.COMPARE_NOT_EQUAL) { - return visitTagEquals(bin.getLeftExpression(), bin.getRightExpression(), tagsVar, true); - } - if (opType == Types.LOGICAL_OR) { - return visitFilterCondition(bin.getLeftExpression(), tagsVar) - + " || " + visitFilterCondition(bin.getRightExpression(), tagsVar); - } - if (opType == Types.LOGICAL_AND) { - return visitFilterCondition(bin.getLeftExpression(), tagsVar) - + " && " + visitFilterCondition(bin.getRightExpression(), tagsVar); - } - if (opType == Types.KEYWORD_IN) { - final String val = visitTagValue(bin.getLeftExpression(), tagsVar); - final String list = visitTagValue(bin.getRightExpression(), tagsVar); - return list + ".contains(" + val + ")"; - } - } - - if (expr instanceof BooleanExpression) { - return visitFilterCondition(((BooleanExpression) expr).getExpression(), tagsVar); - } - - final String val = visitTagValue(expr, tagsVar); - return "(" + val + " != null && !" + val + ".isEmpty())"; - } - - private String visitTagExpr(final Expression expr, final String tagsVar) { - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - if (bin.getOperation().getType() == Types.ASSIGN) { - return visitTagAssignment(bin.getLeftExpression(), bin.getRightExpression(), tagsVar); - } - } - if (expr instanceof MethodCallExpression) { - final MethodCallExpression mce = (MethodCallExpression) expr; - if ("remove".equals(mce.getMethodAsString()) && isTagsVar(mce.getObjectExpression(), tagsVar)) { - final ArgumentListExpression args = toArgList(mce.getArguments()); - return tagsVar + ".remove(" + visitTagValue(args.getExpression(0), tagsVar) + ")"; - } - } - return visitTagValue(expr, tagsVar); - } - - private String visitTagAssignment(final Expression left, final Expression right, final String tagsVar) { - final String val = visitTagValue(right, tagsVar); - - if (left instanceof PropertyExpression) { - final PropertyExpression prop = (PropertyExpression) left; - if (isTagsVar(prop.getObjectExpression(), tagsVar)) { - return tagsVar + ".put(\"" + escapeJava(prop.getPropertyAsString()) + "\", " + val + ")"; - } - } - if (left instanceof BinaryExpression) { - final BinaryExpression sub = (BinaryExpression) left; - if (sub.getOperation().getType() == Types.LEFT_SQUARE_BRACKET - && isTagsVar(sub.getLeftExpression(), tagsVar)) { - final String key = visitTagValue(sub.getRightExpression(), tagsVar); - return tagsVar + ".put(" + key + ", " + val + ")"; - } - } - throw new UnsupportedOperationException( - "Unsupported tag assignment target: " + left.getClass().getSimpleName() + " = " + left.getText()); - } - - String visitTagValue(final Expression expr, final String tagsVar) { - if (expr instanceof PropertyExpression) { - final PropertyExpression prop = (PropertyExpression) expr; - if (isTagsVar(prop.getObjectExpression(), tagsVar)) { - return tagsVar + ".get(\"" + escapeJava(prop.getPropertyAsString()) + "\")"; - } - return visitProperty(prop); - } - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - if (bin.getOperation().getType() == Types.LEFT_SQUARE_BRACKET - && isTagsVar(bin.getLeftExpression(), tagsVar)) { - return tagsVar + ".get(" + visitTagValue(bin.getRightExpression(), tagsVar) + ")"; - } - if (bin.getOperation().getType() == Types.PLUS) { - return visitTagValue(bin.getLeftExpression(), tagsVar) - + " + " + visitTagValue(bin.getRightExpression(), tagsVar); - } - } - // Elvis operator — must check BEFORE TernaryExpression since it extends it - if (expr instanceof ElvisOperatorExpression) { - final ElvisOperatorExpression elvis = (ElvisOperatorExpression) expr; - final String val = visitTagValue(elvis.getTrueExpression(), tagsVar); - final String defaultVal = visitTagValue(elvis.getFalseExpression(), tagsVar); - return "(" + val + " != null ? " + val + " : " + defaultVal + ")"; - } - if (expr instanceof TernaryExpression) { - final TernaryExpression tern = (TernaryExpression) expr; - final String cond = visitFilterCondition(tern.getBooleanExpression().getExpression(), tagsVar); - final String trueVal = visitTagValue(tern.getTrueExpression(), tagsVar); - final String falseVal = visitTagValue(tern.getFalseExpression(), tagsVar); - return "(" + cond + " ? " + trueVal + " : " + falseVal + ")"; - } - if (expr instanceof MethodCallExpression) { - final MethodCallExpression mce = (MethodCallExpression) expr; - final String obj = visitTagValue(mce.getObjectExpression(), tagsVar); - final ArgumentListExpression args = toArgList(mce.getArguments()); - final List argStrs = new ArrayList<>(); - for (Expression a : args.getExpressions()) { - argStrs.add(visitTagValue(a, tagsVar)); - } - final String call = obj + "." + mce.getMethodAsString() + "(" + String.join(", ", argStrs) + ")"; - if (mce.isSafe()) { - return "(" + obj + " != null ? " + call + " : null)"; - } - return call; - } - if (expr instanceof VariableExpression) { - final String name = ((VariableExpression) expr).getName(); - if (name.equals(tagsVar)) { - return tagsVar; - } - return name; - } - if (expr instanceof ConstantExpression) { - return visitConstant((ConstantExpression) expr); - } - if (expr instanceof ListExpression) { - return visitList((ListExpression) expr); - } - if (expr instanceof MapExpression) { - final MapExpression map = (MapExpression) expr; - final List entries = new ArrayList<>(); - for (MapEntryExpression entry : map.getMapEntryExpressions()) { - entries.add(visitTagValue(entry.getKeyExpression(), tagsVar)); - entries.add(visitTagValue(entry.getValueExpression(), tagsVar)); - } - return "Map.of(" + String.join(", ", entries) + ")"; - } - return visitExpression(expr); - } - - private boolean isTagsVar(final Expression expr, final String tagsVar) { - return expr instanceof VariableExpression - && ((VariableExpression) expr).getName().equals(tagsVar); - } - - private List getStatements(final Statement stmt) { - if (stmt instanceof BlockStatement) { - return ((BlockStatement) stmt).getStatements(); - } - return List.of(stmt); - } - - // ---- forEach() Closure ---- - - private String visitForEachClosure(final ClosureExpression closure) { - final Parameter[] params = closure.getParameters(); - final String prefixVar = (params != null && params.length > 0) ? params[0].getName() : "prefix"; - final String tagsVar = (params != null && params.length > 1) ? params[1].getName() : "tags"; - - final List stmts = getStatements(closure.getCode()); - - final StringBuilder sb = new StringBuilder(); - sb.append("(").append(prefixVar).append(", ").append(tagsVar).append(") -> {\n"); - for (Statement s : stmts) { - sb.append(" ").append(visitForEachStatement(s, tagsVar)).append("\n"); - } - sb.append(" }"); - return sb.toString(); - } - - private String visitForEachStatement(final Statement stmt, final String tagsVar) { - if (stmt instanceof ExpressionStatement) { - return visitForEachExpr(((ExpressionStatement) stmt).getExpression(), tagsVar) + ";"; - } - if (stmt instanceof ReturnStatement) { - return "return;"; - } - if (stmt instanceof IfStatement) { - return visitForEachIf((IfStatement) stmt, tagsVar); - } - throw new UnsupportedOperationException( - "Unsupported forEach closure statement: " + stmt.getClass().getSimpleName()); - } - - private String visitForEachExpr(final Expression expr, final String tagsVar) { - if (expr instanceof DeclarationExpression) { - final DeclarationExpression decl = (DeclarationExpression) expr; - final String typeName = decl.getVariableExpression().getType().getNameWithoutPackage(); - final String varName = decl.getVariableExpression().getName(); - final String init = visitTagValue(decl.getRightExpression(), tagsVar); - return typeName + " " + varName + " = " + init; - } - if (expr instanceof BinaryExpression) { - final BinaryExpression bin = (BinaryExpression) expr; - if (bin.getOperation().getType() == Types.ASSIGN) { - final Expression left = bin.getLeftExpression(); - if (isTagWrite(left, tagsVar)) { - return visitTagAssignment(left, bin.getRightExpression(), tagsVar); - } - if (left instanceof VariableExpression) { - return ((VariableExpression) left).getName() - + " = " + visitTagValue(bin.getRightExpression(), tagsVar); - } - } - } - return visitTagExpr(expr, tagsVar); - } - - private String visitForEachIf(final IfStatement ifStmt, final String tagsVar) { - final String condition = visitTagCondition(ifStmt.getBooleanExpression().getExpression(), tagsVar); - final List ifBody = getStatements(ifStmt.getIfBlock()); - final Statement elseBlock = ifStmt.getElseBlock(); - - final StringBuilder sb = new StringBuilder(); - sb.append("if (").append(condition).append(") {\n"); - for (Statement s : ifBody) { - sb.append(" ").append(visitForEachStatement(s, tagsVar)).append("\n"); - } - sb.append(" }"); - - if (elseBlock instanceof IfStatement) { - sb.append(" else ").append(visitForEachIf((IfStatement) elseBlock, tagsVar)); - } else if (elseBlock != null && !(elseBlock instanceof EmptyStatement)) { - sb.append(" else {\n"); - for (Statement s : getStatements(elseBlock)) { - sb.append(" ").append(visitForEachStatement(s, tagsVar)).append("\n"); - } - sb.append(" }"); - } - - return sb.toString(); - } - - private boolean isTagWrite(final Expression left, final String tagsVar) { - if (left instanceof PropertyExpression) { - return isTagsVar(((PropertyExpression) left).getObjectExpression(), tagsVar); - } - if (left instanceof BinaryExpression) { - final BinaryExpression sub = (BinaryExpression) left; - return sub.getOperation().getType() == Types.LEFT_SQUARE_BRACKET - && isTagsVar(sub.getLeftExpression(), tagsVar); - } - return false; - } - - private boolean isNullConstant(final Expression expr) { - return expr instanceof ConstantExpression && ((ConstantExpression) expr).getValue() == null; - } - - // ---- PropertiesExtractor Closure ---- - - private String visitPropertiesExtractorClosure(final ClosureExpression closure) { - final Parameter[] params = closure.getParameters(); - final String tagsVar = (params != null && params.length > 0) ? params[0].getName() : "tags"; - final List stmts = getStatements(closure.getCode()); - - final Statement last = stmts.get(stmts.size() - 1); - Expression bodyExpr; - if (last instanceof ExpressionStatement) { - bodyExpr = ((ExpressionStatement) last).getExpression(); - } else if (last instanceof ReturnStatement) { - bodyExpr = ((ReturnStatement) last).getExpression(); - } else { - throw new UnsupportedOperationException( - "Unsupported PropertiesExtractor closure body: " + last.getClass().getSimpleName()); - } - return "(" + tagsVar + " -> " + visitTagValue(bodyExpr, tagsVar) + ")"; - } - - // ---- Batch Registration, Compilation, and Manifest Writing ---- - - public void registerExpression(final String className, final String source) { - expressionSources.put(className, source); - } - - public void registerFilter(final String className, final String filterLiteral, final String source) { - filterSources.put(className, source); - filterLiteralToClass.put(filterLiteral, GENERATED_PACKAGE + "." + className); - } - - /** - * Compile all registered sources using javax.tools.JavaCompiler. - * - * @param sourceDir directory to write .java source files (package dirs created automatically) - * @param outputDir directory for compiled .class files - * @param classpath classpath for javac (semicolon/colon-separated JAR paths) - * @throws IOException if file I/O fails - */ - public void compileAll(final File sourceDir, final File outputDir, - final String classpath) throws IOException { - final Map allSources = new LinkedHashMap<>(); - allSources.putAll(expressionSources); - allSources.putAll(filterSources); - - if (allSources.isEmpty()) { - log.info("No MAL sources to compile."); - return; - } - - final String packageDir = GENERATED_PACKAGE.replace('.', File.separatorChar); - final File srcPkgDir = new File(sourceDir, packageDir); - if (!srcPkgDir.exists() && !srcPkgDir.mkdirs()) { - throw new IOException("Failed to create source dir: " + srcPkgDir); - } - if (!outputDir.exists() && !outputDir.mkdirs()) { - throw new IOException("Failed to create output dir: " + outputDir); - } - - final List javaFiles = new ArrayList<>(); - for (Map.Entry entry : allSources.entrySet()) { - final File javaFile = new File(srcPkgDir, entry.getKey() + ".java"); - Files.writeString(javaFile.toPath(), entry.getValue()); - javaFiles.add(javaFile); - } - - final JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); - if (compiler == null) { - throw new IllegalStateException("No Java compiler available — requires JDK"); - } - - final StringWriter errorWriter = new StringWriter(); - - try (StandardJavaFileManager fileManager = compiler.getStandardFileManager(null, null, null)) { - final Iterable compilationUnits = - fileManager.getJavaFileObjectsFromFiles(javaFiles); - - final List options = Arrays.asList( - "-d", outputDir.getAbsolutePath(), - "-classpath", classpath - ); - - final JavaCompiler.CompilationTask task = compiler.getTask( - errorWriter, fileManager, null, options, null, compilationUnits); - - final boolean success = task.call(); - if (!success) { - throw new RuntimeException( - "Java compilation failed for " + javaFiles.size() + " MAL sources:\n" - + errorWriter); - } - } - - log.info("Compiled {} MAL sources to {}", allSources.size(), outputDir); - } - - /** - * Write mal-expressions.txt manifest: one FQCN per line. - */ - public void writeExpressionManifest(final File outputDir) throws IOException { - final File manifestDir = new File(outputDir, "META-INF"); - if (!manifestDir.exists() && !manifestDir.mkdirs()) { - throw new IOException("Failed to create META-INF dir: " + manifestDir); - } - - final List lines = expressionSources.keySet().stream() - .map(name -> GENERATED_PACKAGE + "." + name) - .collect(Collectors.toList()); - Files.write(new File(manifestDir, "mal-expressions.txt").toPath(), lines); - log.info("Wrote mal-expressions.txt with {} entries", lines.size()); - } - - /** - * Write mal-filter-expressions.properties manifest: literal=FQCN. - */ - public void writeFilterManifest(final File outputDir) throws IOException { - final File manifestDir = new File(outputDir, "META-INF"); - if (!manifestDir.exists() && !manifestDir.mkdirs()) { - throw new IOException("Failed to create META-INF dir: " + manifestDir); - } - - final List lines = filterLiteralToClass.entrySet().stream() - .map(e -> escapeProperties(e.getKey()) + "=" + e.getValue()) - .collect(Collectors.toList()); - Files.write(new File(manifestDir, "mal-filter-expressions.properties").toPath(), lines); - log.info("Wrote mal-filter-expressions.properties with {} entries", lines.size()); - } - - private static String escapeProperties(final String s) { - return s.replace("\\", "\\\\") - .replace("=", "\\=") - .replace(":", "\\:") - .replace(" ", "\\ "); - } - - // ---- Argument Utilities ---- - - private ArgumentListExpression toArgList(final Expression args) { - if (args instanceof ArgumentListExpression) { - return (ArgumentListExpression) args; - } - if (args instanceof TupleExpression) { - final ArgumentListExpression ale = new ArgumentListExpression(); - for (Expression e : ((TupleExpression) args).getExpressions()) { - ale.addExpression(e); - } - return ale; - } - final ArgumentListExpression ale = new ArgumentListExpression(); - ale.addExpression(args); - return ale; - } - - private List visitArgList(final ArgumentListExpression args) { - final List result = new ArrayList<>(); - for (Expression arg : args.getExpressions()) { - result.add(visitExpression(arg)); - } - return result; - } - - // ---- Utility ---- - - public static String escapeJava(final String s) { - return s.replace("\\", "\\\\") - .replace("\"", "\\\"") - .replace("\n", "\\n") - .replace("\r", "\\r") - .replace("\t", "\\t"); - } -} diff --git a/oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java b/oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java deleted file mode 100644 index 0207d8df32d2..000000000000 --- a/oap-server/analyzer/mal-transpiler/src/test/java/org/apache/skywalking/oap/server/transpiler/mal/MalToJavaTranspilerTest.java +++ /dev/null @@ -1,904 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.server.transpiler.mal; - -import java.io.File; -import java.nio.file.Files; -import java.nio.file.Path; -import java.util.List; - -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; - -import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertTrue; - -class MalToJavaTranspilerTest { - - private MalToJavaTranspiler transpiler; - - @BeforeEach - void setUp() { - transpiler = new MalToJavaTranspiler(); - } - - // ---- AST Parsing + Simple Variable References ---- - - @Test - void simpleVariableReference() { - final String java = transpiler.transpileExpression("MalExpr_test", "metric_name"); - assertNotNull(java); - - assertTrue(java.contains("package " + MalToJavaTranspiler.GENERATED_PACKAGE), - "Should have correct package"); - assertTrue(java.contains("public class MalExpr_test implements MalExpression"), - "Should implement MalExpression"); - assertTrue(java.contains("public SampleFamily run(Map samples)"), - "Should have run method"); - - assertTrue(java.contains("ctx.getSamples().add(\"metric_name\")"), - "Should track sample name in parsing context"); - - assertTrue(java.contains("samples.getOrDefault(\"metric_name\", SampleFamily.EMPTY)"), - "Should look up sample family from map"); - } - - @Test - void downsamplingConstantNotTrackedAsSample() { - final String java = transpiler.transpileExpression("MalExpr_test", "SUM"); - assertNotNull(java); - - assertTrue(!java.contains("ctx.getSamples().add(\"SUM\")"), - "Should not track DownsamplingType constant as sample"); - - assertTrue(java.contains("DownsamplingType.SUM"), - "Should resolve to DownsamplingType.SUM"); - } - - @Test - void parseToAST_producesModuleNode() { - final var ast = transpiler.parseToAST("some_metric"); - assertNotNull(ast, "Should produce a ModuleNode"); - assertNotNull(ast.getStatementBlock(), "Should have a statement block"); - } - - @Test - void constantString() { - final String java = transpiler.transpileExpression("MalExpr_test", "'hello'"); - assertNotNull(java); - assertTrue(java.contains("\"hello\""), - "Should convert Groovy string to Java string"); - } - - @Test - void constantNumber() { - final String java = transpiler.transpileExpression("MalExpr_test", "42"); - assertNotNull(java); - assertTrue(java.contains("42"), - "Should preserve number literal"); - } - - // ---- Method Chains + List Literals + Enum Properties ---- - - @Test - void simpleMethodChain() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric_name.sum(['a', 'b']).service(['svc'], Layer.GENERAL)"); - assertNotNull(java); - - assertTrue(java.contains(".sum(List.of(\"a\", \"b\"))"), - "Should translate ['a','b'] to List.of(\"a\", \"b\")"); - assertTrue(java.contains(".service(List.of(\"svc\"), Layer.GENERAL)"), - "Should translate Layer.GENERAL as enum"); - } - - @Test - void tagEqualChain() { - final String java = transpiler.transpileExpression("MalExpr_test", - "cpu_seconds.tagNotEqual('mode', 'idle').sum(['host']).rate('PT1M')"); - assertNotNull(java); - - assertTrue(java.contains(".tagNotEqual(\"mode\", \"idle\")"), - "Should translate tagNotEqual with string args"); - assertTrue(java.contains(".sum(List.of(\"host\"))"), - "Should translate single-element list"); - assertTrue(java.contains(".rate(\"PT1M\")"), - "Should translate rate with string arg"); - } - - @Test - void downsamplingMethod() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.downsampling(SUM)"); - assertNotNull(java); - - assertTrue(java.contains(".downsampling(DownsamplingType.SUM)"), - "Should resolve SUM to DownsamplingType.SUM"); - } - - @Test - void retagByK8sMeta() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.retagByK8sMeta('service', K8sRetagType.Pod2Service, 'pod', 'namespace')"); - assertNotNull(java); - - assertTrue(java.contains(".retagByK8sMeta(\"service\", K8sRetagType.Pod2Service, \"pod\", \"namespace\")"), - "Should translate K8sRetagType enum and string args"); - } - - @Test - void histogramPercentile() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.sum(['le', 'svc']).histogram().histogram_percentile([50, 75, 90])"); - assertNotNull(java); - - assertTrue(java.contains(".histogram()"), - "Should translate no-arg histogram()"); - assertTrue(java.contains(".histogram_percentile(List.of(50, 75, 90))"), - "Should translate integer list"); - } - - @Test - void sampleNameCollectionThroughChain() { - final String java = transpiler.transpileExpression("MalExpr_test", - "my_metric.sum(['a']).service(['svc'], Layer.GENERAL)"); - assertNotNull(java); - - assertTrue(java.contains("ctx.getSamples().add(\"my_metric\")"), - "Should collect sample name from root of method chain"); - assertTrue(!java.contains("ctx.getSamples().add(\"a\")"), - "Should NOT collect 'a' (it's a string constant arg, not a sample)"); - } - - @Test - void detectPointEnum() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.serviceRelation(DetectPoint.CLIENT, ['src'], ['dst'], Layer.MESH_DP)"); - assertNotNull(java); - - assertTrue(java.contains("DetectPoint.CLIENT"), - "Should translate DetectPoint enum"); - assertTrue(java.contains("Layer.MESH_DP"), - "Should translate Layer enum"); - } - - @Test - void enumImportsPresent() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.service(['svc'], Layer.GENERAL)"); - assertNotNull(java); - - assertTrue(java.contains("import org.apache.skywalking.oap.server.core.analysis.Layer;"), - "Should import Layer"); - assertTrue(java.contains("import org.apache.skywalking.oap.server.core.source.DetectPoint;"), - "Should import DetectPoint"); - assertTrue(java.contains("import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType;"), - "Should import K8sRetagType"); - } - - // ---- Binary Arithmetic with Operand-Swap ---- - - @Test - void sfTimesNumber() { - final String java = transpiler.transpileExpression("MalExpr_test", "metric * 100"); - assertNotNull(java); - assertTrue(java.contains(".multiply(100)"), - "SF * N should call .multiply(N)"); - } - - @Test - void sfDivNumber() { - final String java = transpiler.transpileExpression("MalExpr_test", "metric / 1024"); - assertNotNull(java); - assertTrue(java.contains(".div(1024)"), - "SF / N should call .div(N)"); - } - - @Test - void numberMinusSf() { - final String java = transpiler.transpileExpression("MalExpr_test", "100 - metric"); - assertNotNull(java); - assertTrue(java.contains(".minus(100).negative()"), - "N - SF should produce sf.minus(N).negative()"); - } - - @Test - void numberDivSf() { - final String java = transpiler.transpileExpression("MalExpr_test", "1 / metric"); - assertNotNull(java); - assertTrue(java.contains(".newValue(v -> 1 / v)"), - "N / SF should produce sf.newValue(v -> N / v)"); - } - - @Test - void numberPlusSf() { - final String java = transpiler.transpileExpression("MalExpr_test", "10 + metric"); - assertNotNull(java); - assertTrue(java.contains(".plus(10)"), - "N + SF should swap to sf.plus(N)"); - } - - @Test - void numberTimesSf() { - final String java = transpiler.transpileExpression("MalExpr_test", "100 * metric"); - assertNotNull(java); - assertTrue(java.contains(".multiply(100)"), - "N * SF should swap to sf.multiply(N)"); - } - - @Test - void sfMinusSf() { - final String java = transpiler.transpileExpression("MalExpr_test", "mem_total - mem_avail"); - assertNotNull(java); - assertTrue(java.contains("ctx.getSamples().add(\"mem_total\")"), - "Should collect both sample names"); - assertTrue(java.contains("ctx.getSamples().add(\"mem_avail\")"), - "Should collect both sample names"); - assertTrue(java.contains(".minus("), - "SF - SF should call .minus()"); - } - - @Test - void sfDivSfTimesNumber() { - final String java = transpiler.transpileExpression("MalExpr_test", - "used_bytes / max_bytes * 100"); - assertNotNull(java); - assertTrue(java.contains(".div("), - "Should have .div() for SF / SF"); - assertTrue(java.contains(".multiply(100)"), - "Should have .multiply(100) for result * 100"); - } - - @Test - void nestedParenArithmetic() { - final String java = transpiler.transpileExpression("MalExpr_test", - "100 - ((mem_free * 100) / mem_total)"); - assertNotNull(java); - assertTrue(java.contains(".multiply(100)"), - "Should have inner multiply"); - assertTrue(java.contains(".negative()"), - "100 - SF should produce .negative()"); - } - - @Test - void parenthesizedWithMethodChain() { - final String java = transpiler.transpileExpression("MalExpr_test", - "(metric * 100).tagNotEqual('mode', 'idle').sum(['host']).rate('PT1M')"); - assertNotNull(java); - assertTrue(java.contains(".multiply(100)"), - "Should have multiply inside parens"); - assertTrue(java.contains(".tagNotEqual(\"mode\", \"idle\")"), - "Should chain tagNotEqual after parens"); - assertTrue(java.contains(".rate(\"PT1M\")"), - "Should chain rate at the end"); - } - - // ---- tag() Closure — Simple Cases ---- - - @Test - void tagAssignmentWithStringConcat() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.route = 'route/' + tags['route']})"); - assertNotNull(java); - - assertTrue(java.contains(".tag((TagFunction)"), - "Should cast closure to TagFunction"); - assertTrue(java.contains("tags.put(\"route\", \"route/\" + tags.get(\"route\"))"), - "Should translate assignment with string concat and subscript read"); - assertTrue(java.contains("return tags;"), - "Should return tags at end of lambda"); - } - - @Test - void tagRemove() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.remove('condition')})"); - assertNotNull(java); - - assertTrue(java.contains("tags.remove(\"condition\")"), - "Should translate remove call"); - } - - @Test - void tagPropertyToProperty() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.rs_nm = tags.set})"); - assertNotNull(java); - - assertTrue(java.contains("tags.put(\"rs_nm\", tags.get(\"set\"))"), - "Should translate property read on RHS to tags.get()"); - } - - @Test - void tagStringConcatWithPropertyRead() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.cluster = 'es::' + tags.cluster})"); - assertNotNull(java); - - assertTrue(java.contains("tags.put(\"cluster\", \"es::\" + tags.get(\"cluster\"))"), - "Should translate string concat with property read"); - } - - @Test - void tagClosureLambdaStructure() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.x = 'y'})"); - assertNotNull(java); - - assertTrue(java.contains("(tags -> {"), - "Should have lambda opening"); - assertTrue(java.contains("return tags;"), - "Should return tags variable"); - } - - @Test - void tagWithSubscriptWrite() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags['service_name'] = tags['svc']})"); - assertNotNull(java); - - assertTrue(java.contains("tags.put(\"service_name\", tags.get(\"svc\"))"), - "Should translate subscript write and read"); - } - - @Test - void tagChainAfterTag() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.x = 'y'}).sum(['host']).service(['svc'], Layer.GENERAL)"); - assertNotNull(java); - - assertTrue(java.contains(".tag((TagFunction)"), - "Should have tag call"); - assertTrue(java.contains(".sum(List.of(\"host\"))"), - "Should chain sum after tag"); - assertTrue(java.contains(".service(List.of(\"svc\"), Layer.GENERAL)"), - "Should chain service after sum"); - } - - // ---- tag() Closure — if/else + Compound Conditions ---- - - @Test - void ifOnlyWithChainedOr() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['gc'] == 'PS Scavenge' || tags['gc'] == 'Copy' || tags['gc'] == 'ParNew' || tags['gc'] == 'G1 Young Generation') {tags.gc = 'young_gc_count'} })"); - assertNotNull(java); - - assertTrue(java.contains("if (\"PS Scavenge\".equals(tags.get(\"gc\"))"), - "Should translate first == with constant on left for null-safety"); - assertTrue(java.contains("|| \"Copy\".equals(tags.get(\"gc\"))"), - "Should chain || for second comparison"); - assertTrue(java.contains("|| \"ParNew\".equals(tags.get(\"gc\"))"), - "Should chain || for third comparison"); - assertTrue(java.contains("|| \"G1 Young Generation\".equals(tags.get(\"gc\"))"), - "Should chain || for fourth comparison"); - assertTrue(java.contains("tags.put(\"gc\", \"young_gc_count\")"), - "Should translate assignment in if body"); - } - - @Test - void ifElse() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['primary'] == 'true') {tags.primary = 'primary'} else {tags.primary = 'replica'} })"); - assertNotNull(java); - - assertTrue(java.contains("if (\"true\".equals(tags.get(\"primary\"))"), - "Should translate == comparison in condition"); - assertTrue(java.contains("tags.put(\"primary\", \"primary\")"), - "Should translate if-branch assignment"); - assertTrue(java.contains("} else {"), - "Should have else clause"); - assertTrue(java.contains("tags.put(\"primary\", \"replica\")"), - "Should translate else-branch assignment"); - } - - @Test - void ifOnlyNoElse() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['level'] == '1') {tags.level = 'L1 aggregation'} })"); - assertNotNull(java); - - assertTrue(java.contains("if (\"1\".equals(tags.get(\"level\"))"), - "Should translate condition"); - assertTrue(java.contains("tags.put(\"level\", \"L1 aggregation\")"), - "Should translate if-body"); - assertTrue(!java.contains("else"), - "Should NOT have else clause"); - } - - @Test - void notEqualComparison() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['status'] != 'ok') {tags.status = 'error'} })"); - assertNotNull(java); - - assertTrue(java.contains("!\"ok\".equals(tags.get(\"status\"))"), - "Should translate != with negated .equals()"); - } - - @Test - void logicalAndCondition() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['a'] == 'x' && tags['b'] == 'y') {tags.c = 'z'} })"); - assertNotNull(java); - - assertTrue(java.contains("\"x\".equals(tags.get(\"a\")) && \"y\".equals(tags.get(\"b\"))"), - "Should translate && with .equals() on both sides"); - } - - @Test - void chainedTagClosuresWithIf() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['level'] == '1') {tags.level = 'L1 aggregation'} })" + - ".tag({tags -> if (tags['level'] == '2') {tags.level = 'L2 aggregation'} })"); - assertNotNull(java); - - assertTrue(java.contains("\"1\".equals(tags.get(\"level\"))"), - "Should translate first tag closure condition"); - assertTrue(java.contains("\"2\".equals(tags.get(\"level\"))"), - "Should translate second tag closure condition"); - } - - @Test - void ifWithMethodChainAfter() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> if (tags['gc'] == 'Copy') {tags.gc = 'young'} }).sum(['host']).service(['svc'], Layer.GENERAL)"); - assertNotNull(java); - - assertTrue(java.contains(".tag((TagFunction)"), - "Should have TagFunction cast"); - assertTrue(java.contains("\"Copy\".equals(tags.get(\"gc\"))"), - "Should have if condition"); - assertTrue(java.contains(".sum(List.of(\"host\"))"), - "Should chain sum after tag"); - } - - // ---- Filter Closures ---- - - @Test - void simpleEqualityFilter() { - final String java = transpiler.transpileFilter("MalFilter_0", - "{ tags -> tags.job_name == 'vm-monitoring' }"); - assertNotNull(java); - - assertTrue(java.contains("public class MalFilter_0 implements MalFilter"), - "Should implement MalFilter"); - assertTrue(java.contains("public boolean test(Map tags)"), - "Should have test method"); - assertTrue(java.contains("\"vm-monitoring\".equals(tags.get(\"job_name\"))"), - "Should translate == with constant on left"); - } - - @Test - void filterPackageAndImports() { - final String java = transpiler.transpileFilter("MalFilter_0", - "{ tags -> tags.job_name == 'x' }"); - assertNotNull(java); - - assertTrue(java.contains("package " + MalToJavaTranspiler.GENERATED_PACKAGE), - "Should have correct package"); - assertTrue(java.contains("import java.util.*;"), - "Should import java.util"); - assertTrue(java.contains("import org.apache.skywalking.oap.meter.analyzer.dsl.*;"), - "Should import dsl package"); - } - - @Test - void orFilter() { - final String java = transpiler.transpileFilter("MalFilter_1", - "{ tags -> tags.job_name == 'flink-jobManager-monitoring' || tags.job_name == 'flink-taskManager-monitoring' }"); - assertNotNull(java); - - assertTrue(java.contains("\"flink-jobManager-monitoring\".equals(tags.get(\"job_name\"))"), - "Should translate first =="); - assertTrue(java.contains("|| \"flink-taskManager-monitoring\".equals(tags.get(\"job_name\"))"), - "Should translate || with second =="); - } - - @Test - void inListFilter() { - final String java = transpiler.transpileFilter("MalFilter_2", - "{ tags -> tags.job_name in ['kubernetes-cadvisor', 'kube-state-metrics'] }"); - assertNotNull(java); - - assertTrue(java.contains("List.of(\"kubernetes-cadvisor\", \"kube-state-metrics\").contains(tags.get(\"job_name\"))"), - "Should translate 'in' to List.of().contains()"); - } - - @Test - void compoundAndFilter() { - final String java = transpiler.transpileFilter("MalFilter_3", - "{ tags -> tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3' }"); - assertNotNull(java); - - assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), - "Should translate first =="); - assertTrue(java.contains("&& \"AWS/S3\".equals(tags.get(\"Namespace\"))"), - "Should translate && with second =="); - } - - @Test - void truthinessFilter() { - final String java = transpiler.transpileFilter("MalFilter_4", - "{ tags -> tags.cloud_provider == 'aws' && tags.Stage }"); - assertNotNull(java); - - assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), - "Should translate =="); - assertTrue(java.contains("(tags.get(\"Stage\") != null && !tags.get(\"Stage\").isEmpty())"), - "Should translate bare tags.Stage as truthiness check"); - } - - @Test - void negatedTruthinessFilter() { - final String java = transpiler.transpileFilter("MalFilter_5", - "{ tags -> tags.cloud_provider == 'aws' && !tags.Method }"); - assertNotNull(java); - - assertTrue(java.contains("(tags.get(\"Method\") == null || tags.get(\"Method\").isEmpty())"), - "Should translate !tags.Method as negated truthiness"); - } - - @Test - void compoundWithTruthinessAndNegation() { - final String java = transpiler.transpileFilter("MalFilter_6", - "{ tags -> tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/ApiGateway' && tags.Stage && !tags.Method }"); - assertNotNull(java); - - assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), - "Should translate first =="); - assertTrue(java.contains("\"AWS/ApiGateway\".equals(tags.get(\"Namespace\"))"), - "Should translate second =="); - assertTrue(java.contains("(tags.get(\"Stage\") != null && !tags.get(\"Stage\").isEmpty())"), - "Should translate truthiness"); - assertTrue(java.contains("(tags.get(\"Method\") == null || tags.get(\"Method\").isEmpty())"), - "Should translate negated truthiness"); - } - - @Test - void wrappedBlockFilter() { - final String java = transpiler.transpileFilter("MalFilter_7", - "{ tags -> {tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3'} }"); - assertNotNull(java); - - assertTrue(java.contains("\"aws\".equals(tags.get(\"cloud_provider\"))"), - "Should unwrap inner block and translate =="); - assertTrue(java.contains("\"AWS/S3\".equals(tags.get(\"Namespace\"))"), - "Should translate second == after unwrapping"); - } - - @Test - void truthinessWithOrInParens() { - final String java = transpiler.transpileFilter("MalFilter_8", - "{ tags -> tags.cloud_provider == 'aws' && (tags.ApiId || tags.ApiName) }"); - assertNotNull(java); - - assertTrue(java.contains("(tags.get(\"ApiId\") != null && !tags.get(\"ApiId\").isEmpty())"), - "Should translate ApiId truthiness"); - assertTrue(java.contains("(tags.get(\"ApiName\") != null && !tags.get(\"ApiName\").isEmpty())"), - "Should translate ApiName truthiness"); - } - - // ---- forEach() Closure ---- - - @Test - void forEachBasicStructure() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['client', 'server'], { prefix, tags -> tags[prefix + '_id'] = 'test' })"); - assertNotNull(java); - - assertTrue(java.contains(".forEach(List.of(\"client\", \"server\"), (ForEachFunction)"), - "Should cast closure to ForEachFunction"); - assertTrue(java.contains("(prefix, tags) -> {"), - "Should have two-parameter lambda"); - assertTrue(java.contains("tags.put(prefix + \"_id\", \"test\")"), - "Should translate dynamic subscript write"); - } - - @Test - void forEachNullCheckWithEarlyReturn() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['client'], { prefix, tags -> if (tags[prefix + '_process_id'] != null) { return } })"); - assertNotNull(java); - - assertTrue(java.contains("tags.get(prefix + \"_process_id\") != null"), - "Should translate null check"); - assertTrue(java.contains("return;"), - "Should have void return for early exit"); - } - - @Test - void forEachWithProcessRegistry() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['client'], { prefix, tags -> " + - "tags[prefix + '_process_id'] = ProcessRegistry.generateVirtualLocalProcess(tags.service, tags.instance) })"); - assertNotNull(java); - - assertTrue(java.contains("ProcessRegistry.generateVirtualLocalProcess(tags.get(\"service\"), tags.get(\"instance\"))"), - "Should translate static method call with tag reads as args"); - assertTrue(java.contains("tags.put(prefix + \"_process_id\","), - "Should translate dynamic subscript write"); - } - - @Test - void forEachVarDeclaration() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['component'], { key, tags -> String result = '' })"); - assertNotNull(java); - - assertTrue(java.contains("String result = \"\""), - "Should translate variable declaration with empty string"); - } - - @Test - void forEachVarDeclWithTagRead() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['component'], { key, tags -> String protocol = tags['protocol'] })"); - assertNotNull(java); - - assertTrue(java.contains("String protocol = tags.get(\"protocol\")"), - "Should translate var decl with tag read"); - } - - @Test - void forEachIfElseIfChain() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['component'], { key, tags -> " + - "String protocol = tags['protocol']\n" + - "String ssl = tags['is_ssl']\n" + - "String result = ''\n" + - "if (protocol == 'http' && ssl == 'true') { result = '129' } " + - "else if (protocol == 'http') { result = '49' } " + - "else if (ssl == 'true') { result = '130' } " + - "else { result = '110' }\n" + - "tags[key] = result })"); - assertNotNull(java); - - assertTrue(java.contains("String protocol = tags.get(\"protocol\")"), - "Should declare protocol"); - assertTrue(java.contains("String ssl = tags.get(\"is_ssl\")"), - "Should declare ssl"); - - assertTrue(java.contains("\"http\".equals(protocol)"), - "Should compare local var with .equals()"); - assertTrue(java.contains("\"true\".equals(ssl)"), - "Should compare ssl with .equals()"); - - assertTrue(java.contains("} else if ("), - "Should produce else-if, not nested else { if }"); - - assertTrue(java.contains("} else {"), - "Should have final else"); - assertTrue(java.contains("result = \"110\""), - "Should assign default value in else"); - - assertTrue(java.contains("tags.put(key, result)"), - "Should write result to tags[key]"); - } - - @Test - void forEachLocalVarAssignment() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['x'], { key, tags -> " + - "String r = ''\n" + - "r = 'abc'\n" + - "tags[key] = r })"); - assertNotNull(java); - - assertTrue(java.contains("r = \"abc\""), - "Should translate local var reassignment"); - } - - @Test - void forEachEqualsOnStringComparison() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['client'], { prefix, tags -> " + - "if (tags[prefix + '_local'] == 'true') { tags[prefix + '_id'] = 'local' } })"); - assertNotNull(java); - - assertTrue(java.contains("\"true\".equals(tags.get(prefix + \"_local\"))"), - "Should translate dynamic subscript comparison with .equals()"); - assertTrue(java.contains("tags.put(prefix + \"_id\", \"local\")"), - "Should translate dynamic subscript assignment"); - } - - @Test - void chainedForEach() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.forEach(['a'], { k1, tags -> tags[k1] = 'x' })" + - ".forEach(['b'], { k2, tags -> tags[k2] = 'y' })"); - assertNotNull(java); - - assertTrue(java.contains("(ForEachFunction) (k1, tags)"), - "Should have first forEach with k1"); - assertTrue(java.contains("(ForEachFunction) (k2, tags)"), - "Should have second forEach with k2"); - } - - // ---- Elvis (?:), Safe Navigation (?.), Ternary (? :) ---- - - @Test - void safeNavigation() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.svc = tags['skywalking_service']?.trim() })"); - assertNotNull(java); - - assertTrue(java.contains("(tags.get(\"skywalking_service\") != null ? tags.get(\"skywalking_service\").trim() : null)"), - "Should translate ?.trim() to null-checked call"); - } - - @Test - void elvisOperator() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.svc = tags['name'] ?: 'unknown' })"); - assertNotNull(java); - - assertTrue(java.contains("(tags.get(\"name\") != null ? tags.get(\"name\") : \"unknown\")"), - "Should translate ?: to null-check with default"); - } - - @Test - void safeNavPlusElvis() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.service_name = 'APISIX::'+(tags['skywalking_service']?.trim()?:'APISIX') })"); - assertNotNull(java); - - assertTrue(java.contains("tags.get(\"skywalking_service\") != null ? tags.get(\"skywalking_service\").trim() : null"), - "Should have safe nav for trim"); - assertTrue(java.contains("!= null ?") && java.contains(": \"APISIX\""), - "Should have elvis default to APISIX"); - assertTrue(java.contains("\"APISIX::\" + "), - "Should have string prefix concatenation"); - } - - @Test - void ternaryOperator() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.tag({tags -> tags.service_name = tags.ApiId ? 'gw::'+tags.ApiId : 'gw::'+tags.ApiName })"); - assertNotNull(java); - - assertTrue(java.contains("tags.get(\"ApiId\") != null && !tags.get(\"ApiId\").isEmpty()"), - "Should translate ternary condition as truthiness check"); - assertTrue(java.contains("\"gw::\" + tags.get(\"ApiId\")"), - "Should have true branch expression"); - assertTrue(java.contains("\"gw::\" + tags.get(\"ApiName\")"), - "Should have false branch expression"); - } - - @Test - void safeNavInFilterCondition() { - final String java = transpiler.transpileFilter("MalFilter_test", - "{ tags -> tags.job_name == 'eks-monitoring' && tags.Service?.trim() }"); - assertNotNull(java); - - assertTrue(java.contains("\"eks-monitoring\".equals(tags.get(\"job_name\"))"), - "Should translate == comparison"); - assertTrue(java.contains("tags.get(\"Service\") != null ? tags.get(\"Service\").trim() : null"), - "Should have safe nav for Service?.trim()"); - } - - // ---- instance() with PropertiesExtractor, MapExpression ---- - - @Test - void instanceWithPropertiesExtractor() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.instance(['cluster', 'service'], '::', ['pod'], '', Layer.K8S_SERVICE, " + - "{tags -> ['pod': tags.pod, 'namespace': tags.namespace]})"); - assertNotNull(java); - - assertTrue(java.contains(".instance("), - "Should have instance call"); - assertTrue(java.contains("(PropertiesExtractor)"), - "Should cast closure to PropertiesExtractor"); - assertTrue(java.contains("Map.of(\"pod\", tags.get(\"pod\"), \"namespace\", tags.get(\"namespace\"))"), - "Should translate map literal to Map.of()"); - } - - @Test - void mapExpressionInTagValue() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.instance(['svc'], ['inst'], Layer.GENERAL, {tags -> ['key': tags.val]})"); - assertNotNull(java); - - assertTrue(java.contains("Map.of(\"key\", tags.get(\"val\"))"), - "Should translate single-entry map"); - assertTrue(java.contains("(PropertiesExtractor)"), - "Should have PropertiesExtractor cast"); - } - - @Test - void processRelationNoClosures() { - final String java = transpiler.transpileExpression("MalExpr_test", - "metric.processRelation('side', ['service'], ['instance'], " + - "'client_process_id', 'server_process_id', 'component')"); - assertNotNull(java); - - assertTrue(java.contains(".processRelation(\"side\", List.of(\"service\"), List.of(\"instance\"), " + - "\"client_process_id\", \"server_process_id\", \"component\")"), - "Should translate processRelation as regular method call"); - } - - // ---- Compilation + Manifests ---- - - @Test - void sourceWrittenForCompilation(@TempDir Path tempDir) throws Exception { - final String source = transpiler.transpileExpression("MalExpr_compile_test", - "metric.sum(['host']).service(['svc'], Layer.GENERAL)"); - transpiler.registerExpression("MalExpr_compile_test", source); - - final File sourceDir = tempDir.resolve("src").toFile(); - final File outputDir = tempDir.resolve("classes").toFile(); - - try { - transpiler.compileAll(sourceDir, outputDir, System.getProperty("java.class.path")); - - final String classPath = MalToJavaTranspiler.GENERATED_PACKAGE.replace('.', File.separatorChar) - + File.separator + "MalExpr_compile_test.class"; - assertTrue(new File(outputDir, classPath).exists(), - "Compiled .class file should exist"); - } catch (RuntimeException e) { - if (e.getMessage().contains("compilation failed")) { - final String pkgPath = MalToJavaTranspiler.GENERATED_PACKAGE.replace('.', File.separatorChar); - final File javaFile = new File(sourceDir, pkgPath + "/MalExpr_compile_test.java"); - assertTrue(javaFile.exists(), "Source .java file should be written"); - final String written = Files.readString(javaFile.toPath()); - assertTrue(written.contains("implements MalExpression"), - "Written source should implement MalExpression"); - } else { - throw e; - } - } - } - - @Test - void expressionManifest(@TempDir Path tempDir) throws Exception { - transpiler.registerExpression("MalExpr_a", - transpiler.transpileExpression("MalExpr_a", "metric_a")); - transpiler.registerExpression("MalExpr_b", - transpiler.transpileExpression("MalExpr_b", "metric_b")); - - final File outputDir = tempDir.toFile(); - transpiler.writeExpressionManifest(outputDir); - - final File manifest = new File(outputDir, "META-INF/mal-expressions.txt"); - assertTrue(manifest.exists(), "Manifest file should exist"); - - final List lines = Files.readAllLines(manifest.toPath()); - assertTrue(lines.contains(MalToJavaTranspiler.GENERATED_PACKAGE + ".MalExpr_a"), - "Should contain MalExpr_a FQCN"); - assertTrue(lines.contains(MalToJavaTranspiler.GENERATED_PACKAGE + ".MalExpr_b"), - "Should contain MalExpr_b FQCN"); - } - - @Test - void filterManifest(@TempDir Path tempDir) throws Exception { - final String literal = "{ tags -> tags.job == 'x' }"; - transpiler.registerFilter("MalFilter_0", literal, - transpiler.transpileFilter("MalFilter_0", literal)); - - final File outputDir = tempDir.toFile(); - transpiler.writeFilterManifest(outputDir); - - final File manifest = new File(outputDir, "META-INF/mal-filter-expressions.properties"); - assertTrue(manifest.exists(), "Filter manifest should exist"); - - final String content = Files.readString(manifest.toPath()); - assertTrue(content.contains(MalToJavaTranspiler.GENERATED_PACKAGE + ".MalFilter_0"), - "Should contain MalFilter_0 FQCN"); - } -} diff --git a/oap-server/analyzer/meter-analyzer-v2/pom.xml b/oap-server/analyzer/meter-analyzer-v2/pom.xml deleted file mode 100644 index 8bbcaa43ed47..000000000000 --- a/oap-server/analyzer/meter-analyzer-v2/pom.xml +++ /dev/null @@ -1,78 +0,0 @@ - - - - - - analyzer - org.apache.skywalking - ${revision} - - 4.0.0 - - meter-analyzer-v2 - Pure Java MAL runtime that loads transpiled MalExpression/MalFilter classes from manifests instead of Groovy - - - - org.apache.skywalking - meter-analyzer - ${project.version} - - - - - - - org.apache.maven.plugins - maven-shade-plugin - - - package - - shade - - - true - - - org.apache.skywalking:meter-analyzer - - - - - org.apache.skywalking:meter-analyzer - - org/apache/skywalking/oap/meter/analyzer/dsl/DSL.class - org/apache/skywalking/oap/meter/analyzer/dsl/DSL$*.class - org/apache/skywalking/oap/meter/analyzer/dsl/Expression.class - org/apache/skywalking/oap/meter/analyzer/dsl/Expression$*.class - org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.class - org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression$*.class - org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.class - org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure$*.class - - - - - - - - - - diff --git a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java deleted file mode 100644 index 3098ac7672d7..000000000000 --- a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java +++ /dev/null @@ -1,116 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.meter.analyzer.dsl; - -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.nio.charset.StandardCharsets; -import java.util.HashMap; -import java.util.Map; -import java.util.concurrent.atomic.AtomicInteger; -import lombok.extern.slf4j.Slf4j; - -/** - * Same-FQCN replacement for upstream MAL DSL. - * Loads transpiled {@link MalExpression} classes from mal-expressions.txt manifest - * instead of Groovy {@code DelegatingScript} classes -- no Groovy runtime needed. - */ -@Slf4j -public final class DSL { - private static final String MANIFEST_PATH = "META-INF/mal-expressions.txt"; - private static volatile Map SCRIPT_MAP; - private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); - - /** - * Parse string literal to Expression object, which can be reused. - * - * @param metricName the name of metric defined in mal rule - * @param expression string literal represents the DSL expression. - * @return Expression object could be executed. - */ - public static Expression parse(final String metricName, final String expression) { - if (metricName == null) { - throw new UnsupportedOperationException( - "Init expressions (metricName=null) are not supported in v2 mode. " - + "All init expressions must be pre-compiled at build time."); - } - - final Map scriptMap = loadManifest(); - final String className = scriptMap.get(metricName); - if (className == null) { - throw new IllegalStateException( - "Transpiled MAL expression not found for metric: " + metricName - + ". Available: " + scriptMap.size() + " expressions"); - } - - try { - final Class exprClass = Class.forName(className); - final MalExpression malExpr = (MalExpression) exprClass.getDeclaredConstructor().newInstance(); - final int count = LOADED_COUNT.incrementAndGet(); - log.debug("Loaded transpiled MAL expression [{}/{}]: {}", count, scriptMap.size(), metricName); - return new Expression(metricName, expression, malExpr); - } catch (ClassNotFoundException e) { - throw new IllegalStateException( - "Transpiled MAL expression class not found: " + className, e); - } catch (ReflectiveOperationException e) { - throw new IllegalStateException( - "Failed to instantiate transpiled MAL expression: " + className, e); - } - } - - private static Map loadManifest() { - if (SCRIPT_MAP != null) { - return SCRIPT_MAP; - } - synchronized (DSL.class) { - if (SCRIPT_MAP != null) { - return SCRIPT_MAP; - } - final Map map = new HashMap<>(); - try (InputStream is = DSL.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { - if (is == null) { - log.warn("MAL expression manifest not found: {}", MANIFEST_PATH); - SCRIPT_MAP = map; - return map; - } - try (BufferedReader reader = new BufferedReader( - new InputStreamReader(is, StandardCharsets.UTF_8))) { - String line; - while ((line = reader.readLine()) != null) { - line = line.trim(); - if (line.isEmpty()) { - continue; - } - final String simpleName = line.substring(line.lastIndexOf('.') + 1); - if (simpleName.startsWith("MalExpr_")) { - final String metric = simpleName.substring("MalExpr_".length()); - map.put(metric, line); - } - } - } - } catch (IOException e) { - throw new IllegalStateException("Failed to load MAL expression manifest", e); - } - log.info("Loaded {} transpiled MAL expressions from manifest", map.size()); - SCRIPT_MAP = map; - return map; - } - } -} diff --git a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java deleted file mode 100644 index cf2e2083017c..000000000000 --- a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java +++ /dev/null @@ -1,92 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.meter.analyzer.dsl; - -import com.google.common.collect.ImmutableMap; -import java.util.Map; -import lombok.ToString; -import lombok.extern.slf4j.Slf4j; - -/** - * Same-FQCN replacement for upstream Expression. - * Wraps a transpiled {@link MalExpression} (pure Java) instead of a Groovy DelegatingScript. - * No ExpandoMetaClass, no propertyMissing(), no ThreadLocal sample repository. - */ -@Slf4j -@ToString(of = {"literal"}) -public class Expression { - - private final String metricName; - private final String literal; - private final MalExpression expression; - - public Expression(final String metricName, final String literal, final MalExpression expression) { - this.metricName = metricName; - this.literal = literal; - this.expression = expression; - } - - /** - * Parse the expression statically. - * - * @return Parsed context of the expression. - */ - public ExpressionParsingContext parse() { - try (ExpressionParsingContext ctx = ExpressionParsingContext.create()) { - final Result r = run(ImmutableMap.of()); - if (!r.isSuccess() && r.isThrowable()) { - throw new ExpressionParsingException( - "failed to parse expression: " + literal + ", error:" + r.getError()); - } - if (log.isDebugEnabled()) { - log.debug("\"{}\" is parsed", literal); - } - ctx.validate(literal); - return ctx; - } - } - - /** - * Run the expression with a data map. - * - * @param sampleFamilies a data map includes all of candidates to be analysis. - * @return The result of execution. - */ - public Result run(final Map sampleFamilies) { - try { - for (final SampleFamily s : sampleFamilies.values()) { - if (s != SampleFamily.EMPTY) { - s.context.setMetricName(metricName); - } - } - final SampleFamily sf = expression.run(sampleFamilies); - if (sf == SampleFamily.EMPTY) { - if (ExpressionParsingContext.get().isEmpty()) { - if (log.isDebugEnabled()) { - log.debug("result of {} is empty by \"{}\"", sampleFamilies, literal); - } - } - return Result.fail("Parsed result is an EMPTY sample family"); - } - return Result.success(sf); - } catch (Throwable t) { - log.error("failed to run \"{}\"", literal, t); - return Result.fail(t); - } - } -} diff --git a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java b/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java deleted file mode 100644 index b92318d9d372..000000000000 --- a/oap-server/analyzer/meter-analyzer-v2/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java +++ /dev/null @@ -1,113 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package org.apache.skywalking.oap.meter.analyzer.dsl; - -import java.io.IOException; -import java.io.InputStream; -import java.util.HashMap; -import java.util.Map; -import java.util.Objects; -import java.util.Properties; -import java.util.concurrent.atomic.AtomicInteger; -import lombok.ToString; -import lombok.extern.slf4j.Slf4j; - -/** - * Same-FQCN replacement for upstream FilterExpression. - * Loads transpiled {@link MalFilter} classes from mal-filter-expressions.properties - * manifest instead of Groovy filter closures -- no Groovy runtime needed. - */ -@Slf4j -@ToString(of = {"literal"}) -public class FilterExpression { - private static final String MANIFEST_PATH = "META-INF/mal-filter-expressions.properties"; - private static volatile Map FILTER_MAP; - private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); - - private final String literal; - private final MalFilter malFilter; - - @SuppressWarnings("unchecked") - public FilterExpression(final String literal) { - this.literal = literal; - - final Map filterMap = loadManifest(); - final String className = filterMap.get(literal); - if (className == null) { - throw new IllegalStateException( - "Transpiled MAL filter not found for: " + literal - + ". Available filters: " + filterMap.size()); - } - - try { - final Class filterClass = Class.forName(className); - malFilter = (MalFilter) filterClass.getDeclaredConstructor().newInstance(); - final int count = LOADED_COUNT.incrementAndGet(); - log.debug("Loaded transpiled MAL filter [{}/{}]: {}", count, filterMap.size(), literal); - } catch (ClassNotFoundException e) { - throw new IllegalStateException( - "Transpiled MAL filter class not found: " + className, e); - } catch (ReflectiveOperationException e) { - throw new IllegalStateException( - "Failed to instantiate transpiled MAL filter: " + className, e); - } - } - - public Map filter(final Map sampleFamilies) { - try { - final Map result = new HashMap<>(); - for (final Map.Entry entry : sampleFamilies.entrySet()) { - final SampleFamily afterFilter = entry.getValue().filter(malFilter::test); - if (!Objects.equals(afterFilter, SampleFamily.EMPTY)) { - result.put(entry.getKey(), afterFilter); - } - } - return result; - } catch (Throwable t) { - log.error("failed to run \"{}\"", literal, t); - } - return sampleFamilies; - } - - private static Map loadManifest() { - if (FILTER_MAP != null) { - return FILTER_MAP; - } - synchronized (FilterExpression.class) { - if (FILTER_MAP != null) { - return FILTER_MAP; - } - final Map map = new HashMap<>(); - try (InputStream is = FilterExpression.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { - if (is == null) { - log.warn("MAL filter manifest not found: {}", MANIFEST_PATH); - FILTER_MAP = map; - return map; - } - final Properties props = new Properties(); - props.load(is); - props.forEach((k, v) -> map.put((String) k, (String) v)); - } catch (IOException e) { - throw new IllegalStateException("Failed to load MAL filter manifest", e); - } - log.info("Loaded {} transpiled MAL filters from manifest", map.size()); - FILTER_MAP = map; - return map; - } - } -} diff --git a/oap-server/analyzer/meter-analyzer/CLAUDE.md b/oap-server/analyzer/meter-analyzer/CLAUDE.md new file mode 100644 index 000000000000..2cdf9e720b15 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/CLAUDE.md @@ -0,0 +1,100 @@ +# MAL Compiler + +Compiles MAL (Meter Analysis Language) expressions into `MalExpression` implementation classes at runtime using ANTLR4 parsing and Javassist bytecode generation. + +## Compilation Workflow + +``` +MAL expression string + → MALScriptParser.parse(expression) [ANTLR4 lexer/parser → visitor] + → MALExpressionModel.Expr (immutable AST) + → MALClassGenerator.compileFromModel(name, ast) + 1. collectClosures(ast) — pre-scan for closure arguments + 2. compileClosureClass() — generate each closure as a separate Javassist class + 3. classPool.makeClass() — create main class implementing MalExpression + 4. generateRunMethod() — emit Java source for run(Map) + 5. ctClass.toClass(MalExpressionPackageHolder.class) — load via package anchor + 6. wire closure fields via reflection + → MalExpression instance +``` + +The generated class implements `MalExpression`: +```java +SampleFamily run(Map samples) // pure computation, no side effects +ExpressionMetadata metadata() // compile-time metadata from AST +``` + +## File Structure + +``` +oap-server/analyzer/meter-analyzer/ + src/main/antlr4/.../MALLexer.g4 — ANTLR4 lexer grammar + src/main/antlr4/.../MALParser.g4 — ANTLR4 parser grammar + + src/main/java/.../compiler/ + MALScriptParser.java — ANTLR4 facade: expression → AST + MALExpressionModel.java — Immutable AST model classes + MALClassGenerator.java — Javassist code generator + rt/ + MalExpressionPackageHolder.java — Class loading anchor (empty marker) + + src/test/java/.../compiler/ + MALScriptParserTest.java — 14 parser tests + MALClassGeneratorTest.java — 9 generator tests +``` + +## Package & Class Naming + +| Component | Package / Name | +|-----------|---------------| +| Parser/Model/Generator | `org.apache.skywalking.oap.meter.analyzer.compiler` | +| Generated classes | `org.apache.skywalking.oap.meter.analyzer.compiler.rt.MalExpr_` | +| Closure classes | `org.apache.skywalking.oap.meter.analyzer.compiler.rt.MalExpr__Closure` | +| Package holder | `org.apache.skywalking.oap.meter.analyzer.compiler.rt.MalExpressionPackageHolder` | +| Functional interface | `org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression` (in meter-analyzer) | + +`` is a global `AtomicInteger` counter. `` is the closure index within the expression. + +## Javassist Constraints + +- **No anonymous inner classes**: Javassist cannot compile `new Consumer() { ... }` or `new Function() { ... }` in method bodies. Closures are pre-compiled as separate `CtClass` instances implementing `SampleFamilyFunctions$TagFunction`, stored as fields (`_closure0`, `_closure1`, ...) on the main class, and wired via reflection after `toClass()`. +- **No lambda expressions**: Use the separate-class approach above. +- **Inner class notation**: Use `$` not `.` for nested classes (e.g., `SampleFamilyFunctions$TagFunction`). +- **`isPresent()`/`get()` instead of `ifPresent()`**: `ifPresent(Consumer)` would require an anonymous class. Use `Optional.isPresent()` + `Optional.get()` pattern. + +## Example + +**Input**: `instance_jvm_cpu.sum(['service', 'instance'])` + +**Generated `run()` method** (pure computation, no ThreadLocal): +```java +public SampleFamily run(Map samples) { + return ((SampleFamily) samples.getOrDefault("instance_jvm_cpu", SampleFamily.EMPTY)) + .sum(java.util.List.of("service", "instance")); +} +``` + +**Generated `metadata()` method** (returns compile-time facts extracted from AST): +```java +public ExpressionMetadata metadata() { + // samples=["instance_jvm_cpu"], aggregationLabels=["service","instance"], ... + return new ExpressionMetadata(...); +} +``` + +**Input with closure**: `metric.tag({ tags -> tags['k'] = 'v' })` + +Two classes are generated: +1. `MalExpr_0_Closure0` — implements `TagFunction` with `Map apply(Map tags) { tags.put("k", "v"); return tags; }` +2. `MalExpr_0` — implements `MalExpression` with field `_closure0`, method body calls `metric.tag(this._closure0)` + +## ExpressionMetadata (replaces ExpressionParsingContext) + +Metadata is extracted statically from the AST at compile time by `MALClassGenerator.extractMetadata()`. No ThreadLocal, no dry-run execution. The `Analyzer` calls `expression.metadata()` to get sample names, scope type, aggregation labels, downsampling, histogram/percentile info. + +## Dependencies + +All within this module (grammar, compiler, and runtime are merged): +- ANTLR4 grammar → generates lexer/parser at build time +- `MalExpression`, `ExpressionMetadata`, `SampleFamily` — in `dsl` package of this module +- `javassist` — bytecode generation diff --git a/oap-server/analyzer/meter-analyzer/pom.xml b/oap-server/analyzer/meter-analyzer/pom.xml index 945fd061b3a3..57d3fd200e47 100644 --- a/oap-server/analyzer/meter-analyzer/pom.xml +++ b/oap-server/analyzer/meter-analyzer/pom.xml @@ -38,13 +38,37 @@ server-core ${project.version} - - org.apache.groovy - groovy - io.vavr vavr + + org.antlr + antlr4-runtime + + + org.javassist + javassist + + + + + + org.antlr + antlr4-maven-plugin + + true + + + + antlr + + antlr4 + + + + + + diff --git a/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALLexer.g4 b/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALLexer.g4 new file mode 100644 index 000000000000..2466d484d6c2 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALLexer.g4 @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +// Meter Analysis Language lexer +lexer grammar MALLexer; + +@Header {package org.apache.skywalking.mal.rt.grammar;} + +// Operators +PLUS: '+'; +MINUS: '-'; +STAR: '*'; +SLASH: '/'; + +// Comparison +DEQ: '=='; +NEQ: '!='; +AND: '&&'; +OR: '||'; + +// Delimiters +DOT: '.'; +COMMA: ','; +L_PAREN: '('; +R_PAREN: ')'; +L_BRACKET: '['; +R_BRACKET: ']'; +L_BRACE: '{'; +R_BRACE: '}'; +SEMI: ';'; +COLON: ':'; +QUESTION: '?'; +ARROW: '->'; +ASSIGN: '='; +GT: '>'; +LT: '<'; +GTE: '>='; +LTE: '<='; +NOT: '!'; + +// Keywords +IF: 'if'; +ELSE: 'else'; +RETURN: 'return'; +NULL: 'null'; +TRUE: 'true'; +FALSE: 'false'; +IN: 'in'; + +// Literals +NUMBER + : Digit+ ('.' Digit+)? + ; + +STRING + : '\'' (~['\\\r\n] | EscapeSequence)* '\'' + | '"' (~["\\\r\n] | EscapeSequence)* '"' + ; + +// Comments +LINE_COMMENT + : '//' ~[\r\n]* -> channel(HIDDEN) + ; + +BLOCK_COMMENT + : '/*' .*? '*/' -> channel(HIDDEN) + ; + +// Whitespace +WS + : [ \t\r\n]+ -> channel(HIDDEN) + ; + +// Identifiers - must come after keywords +IDENTIFIER + : Letter LetterOrDigit* + ; + +// Fragments +fragment EscapeSequence + : '\\' [btnfr"'\\] + | '\\' ([0-3]? [0-7])? [0-7] + ; + +fragment Digit + : [0-9] + ; + +fragment Letter + : [a-zA-Z_] + ; + +fragment LetterOrDigit + : Letter + | [0-9] + ; diff --git a/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 b/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 new file mode 100644 index 000000000000..59af9b14d03e --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +// Meter Analysis Language parser +// +// Covers MAL expression patterns: +// metric_name.tagEqual("k","v").sum(["tag"]).rate("PT1M").service(["svc"], Layer.GENERAL) +// metric1 + metric2, (metric * 100), metric1.div(metric2) +// tag({tags -> tags.key = "val"}), forEach(["prefix"], {prefix, tags -> ...}) +// .valueEqual(1), .retagByK8sMeta("svc", K8sRetagType.Pod2Service, "pod", "ns") +// .histogram().histogram_percentile([50,75,90,95,99]).downsampling(SUM) +parser grammar MALParser; + +@Header {package org.apache.skywalking.mal.rt.grammar;} + +options { tokenVocab=MALLexer; } + +// ==================== Top-level ==================== + +// A MAL expression: arithmetic tree of postfix-chained metric references +expression + : additiveExpression EOF + ; + +// ==================== Arithmetic ==================== + +additiveExpression + : multiplicativeExpression ((PLUS | MINUS) multiplicativeExpression)* + ; + +multiplicativeExpression + : unaryExpression ((STAR | SLASH) unaryExpression)* + ; + +unaryExpression + : MINUS unaryExpression # unaryNeg + | postfixExpression # unaryPostfix + | NUMBER # unaryNumber + ; + +// ==================== Postfix (method chaining) ==================== + +// primary.method1().method2()... +postfixExpression + : primary (DOT methodCall)* + ; + +primary + : IDENTIFIER // metric name + | functionCall // top-level function: count(metric), topN(...) + | L_PAREN additiveExpression R_PAREN // parenthesized: (metric * 100).sum() + ; + +functionCall + : IDENTIFIER L_PAREN argumentList? R_PAREN + ; + +methodCall + : IDENTIFIER L_PAREN argumentList? R_PAREN + ; + +// ==================== Arguments ==================== + +argumentList + : argument (COMMA argument)* + ; + +argument + : additiveExpression // nested expression (metric ref, number, arithmetic) + | stringList // ["tag1", "tag2"] + | numberList // [50, 75, 90, 95, 99] + | closureExpression // {tags -> ...} + | enumRef // Layer.GENERAL, K8sRetagType.Pod2Service + | STRING // "PT1M", "k8s-key" + | boolLiteral // true, false + ; + +stringList + : L_BRACKET STRING (COMMA STRING)* R_BRACKET + ; + +numberList + : L_BRACKET NUMBER (COMMA NUMBER)* R_BRACKET + ; + +enumRef + : IDENTIFIER DOT IDENTIFIER + ; + +boolLiteral + : TRUE | FALSE + ; + +// ==================== Closure expressions ==================== +// +// Used in tag(), forEach(), and filter expressions: +// { tags -> tags.key = "val" } +// { prefix, tags -> if (tags[prefix + "_process_id"] != null) { ... } } +// { tags -> tags.job_name == 'mysql-monitoring' } +// { tags -> { tags.cloud_provider == 'aws' && tags.Namespace == 'AWS/S3' } } + +closureExpression + : L_BRACE closureParams? ARROW closureBody R_BRACE + ; + +closureParams + : IDENTIFIER (COMMA IDENTIFIER)* + ; + +closureBody + : closureStatement+ + | L_BRACE closureStatement+ R_BRACE // optional extra braces: { tags -> { ... } } + ; + +closureStatement + : ifStatement + | returnStatement + | assignmentStatement + | expressionStatement + ; + +// ==================== Closure statements ==================== + +ifStatement + : IF L_PAREN closureCondition R_PAREN closureBlock + (ELSE ifStatement)? + (ELSE closureBlock)? + ; + +closureBlock + : L_BRACE closureStatement* R_BRACE + ; + +returnStatement + : RETURN closureExpr? SEMI? + ; + +assignmentStatement + : closureFieldAccess ASSIGN closureExpr SEMI? + ; + +expressionStatement + : closureExpr SEMI? + ; + +// ==================== Closure expressions (within closures) ==================== + +closureCondition + : closureConditionOr + ; + +closureConditionOr + : closureConditionAnd (OR closureConditionAnd)* + ; + +closureConditionAnd + : closureConditionPrimary (AND closureConditionPrimary)* + ; + +closureConditionPrimary + : NOT closureConditionPrimary # conditionNot + | closureExpr DEQ closureExpr # conditionEq + | closureExpr NEQ closureExpr # conditionNeq + | closureExpr GT closureExpr # conditionGt + | closureExpr LT closureExpr # conditionLt + | closureExpr GTE closureExpr # conditionGte + | closureExpr LTE closureExpr # conditionLte + | closureExpr IN closureListLiteral # conditionIn + | L_PAREN closureCondition R_PAREN # conditionParen + | closureExpr # conditionExpr + ; + +closureExpr + : closureExpr PLUS closureExpr # closureAdd + | closureExpr MINUS closureExpr # closureSub + | closureExpr STAR closureExpr # closureMul + | closureExpr SLASH closureExpr # closureDiv + | closureExprPrimary # closurePrimary + ; + +closureExprPrimary + : STRING # closureString + | NUMBER # closureNumber + | NULL # closureNull + | boolLiteral # closureBool + | closureMethodChain # closureChain + ; + +closureMethodChain + : closureTarget (DOT closureChainSegment)* (safeNav closureChainSegment)* + ; + +closureTarget + : IDENTIFIER + ; + +closureChainSegment + : IDENTIFIER L_PAREN closureArgList? R_PAREN # chainMethodCall + | IDENTIFIER # chainFieldAccess + | L_BRACKET closureExpr R_BRACKET # chainIndexAccess + ; + +safeNav + : QUESTION DOT + ; + +closureArgList + : closureExpr (COMMA closureExpr)* + ; + +closureFieldAccess + : IDENTIFIER (DOT IDENTIFIER)* (L_BRACKET closureExpr R_BRACKET)? + ; + +closureListLiteral + : L_BRACKET (STRING (COMMA STRING)*)? R_BRACKET + ; diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java index a8b6a6662584..0d698a9b6b98 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java @@ -37,7 +37,7 @@ import org.apache.skywalking.oap.meter.analyzer.dsl.DSL; import org.apache.skywalking.oap.meter.analyzer.dsl.DownsamplingType; import org.apache.skywalking.oap.meter.analyzer.dsl.Expression; -import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionParsingContext; +import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionMetadata; import org.apache.skywalking.oap.meter.analyzer.dsl.FilterExpression; import org.apache.skywalking.oap.meter.analyzer.dsl.Result; import org.apache.skywalking.oap.meter.analyzer.dsl.Sample; @@ -89,7 +89,7 @@ public static Analyzer build(final String metricName, if (!Strings.isNullOrEmpty(filterExpression)) { filter = new FilterExpression(filterExpression); } - ExpressionParsingContext ctx = e.parse(); + ExpressionMetadata ctx = e.parse(); Analyzer analyzer = new Analyzer(metricName, filter, e, meterSystem, ctx); analyzer.init(); return analyzer; @@ -105,7 +105,7 @@ public static Analyzer build(final String metricName, private final MeterSystem meterSystem; - private final ExpressionParsingContext ctx; + private final ExpressionMetadata ctx; private MetricType metricType; diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java new file mode 100644 index 000000000000..0d42dd12aaf8 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java @@ -0,0 +1,971 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.compiler; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Set; +import java.util.concurrent.atomic.AtomicInteger; +import javassist.ClassPool; +import javassist.CtClass; +import javassist.CtNewConstructor; +import javassist.CtNewMethod; +import org.apache.skywalking.oap.meter.analyzer.compiler.rt.MalExpressionPackageHolder; +import org.apache.skywalking.oap.meter.analyzer.dsl.DownsamplingType; +import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionMetadata; +import org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +/** + * Generates {@link MalExpression} implementation classes from + * {@link MALExpressionModel} AST using Javassist bytecode generation. + * + *

Each generated class implements: + * 

+ *   SampleFamily run(Map<String, SampleFamily> samples)
+ *
+ */ +public final class MALClassGenerator { + + private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(0); + + private static final String PACKAGE_PREFIX = + "org.apache.skywalking.oap.meter.analyzer.compiler.rt."; + + private static final String SF = "org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily"; + + private static final String ENUM_PACKAGE_PREFIX = + "org.apache.skywalking.oap."; + + /** + * Well-known enum types used in MAL expressions. + */ + private static final java.util.Map ENUM_FQCN; + + static { + ENUM_FQCN = new java.util.HashMap<>(); + ENUM_FQCN.put("Layer", "org.apache.skywalking.oap.server.core.analysis.Layer"); + ENUM_FQCN.put("DetectPoint", "org.apache.skywalking.oap.server.core.source.DetectPoint"); + ENUM_FQCN.put("K8sRetagType", + "org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType"); + ENUM_FQCN.put("DownsamplingType", + "org.apache.skywalking.oap.meter.analyzer.dsl.DownsamplingType"); + } + + private final ClassPool classPool; + private int closureCounter; + + public MALClassGenerator() { + this(ClassPool.getDefault()); + } + + public MALClassGenerator(final ClassPool classPool) { + this.classPool = classPool; + } + + /** + * Compiles a MAL expression into a MalExpression implementation. + * + * @param metricName the metric name (used in the generated class name) + * @param expression the MAL expression string + * @return a MalExpression instance + */ + public MalExpression compile(final String metricName, + final String expression) throws Exception { + final MALExpressionModel.Expr ast = MALScriptParser.parse(expression); + return compileFromModel(metricName, ast); + } + + /** + * Compiles from a pre-parsed AST model. + */ + public MalExpression compileFromModel(final String metricName, + final MALExpressionModel.Expr ast) throws Exception { + final String className = PACKAGE_PREFIX + "MalExpr_" + + CLASS_COUNTER.getAndIncrement(); + + closureCounter = 0; + final List closures = new ArrayList<>(); + collectClosures(ast, closures); + + // Generate closure classes first + final List closureInstances = new ArrayList<>(); + for (int i = 0; i < closures.size(); i++) { + final String closureName = className + "_Closure" + i; + final Object instance = compileClosureClass( + closureName, closures.get(i)); + closureInstances.add(instance); + } + + final CtClass ctClass = classPool.makeClass(className); + ctClass.addInterface(classPool.get( + "org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression")); + + // Add closure fields + for (int i = 0; i < closures.size(); i++) { + final String fieldType = closures.get(i).interfaceType; + ctClass.addField(javassist.CtField.make( + "public " + fieldType + " _closure" + i + ";", ctClass)); + } + + ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); + + final String runBody = generateRunMethod(ast); + ctClass.addMethod(CtNewMethod.make(runBody, ctClass)); + + final ExpressionMetadata metadata = extractMetadata(ast); + final String metadataBody = generateMetadataMethod(metadata); + ctClass.addMethod(CtNewMethod.make(metadataBody, ctClass)); + + final Class clazz = ctClass.toClass(MalExpressionPackageHolder.class); + ctClass.detach(); + final MalExpression instance = (MalExpression) clazz.getDeclaredConstructor() + .newInstance(); + + // Set closure fields via reflection + for (int i = 0; i < closureInstances.size(); i++) { + final java.lang.reflect.Field field = + clazz.getField("_closure" + i); + field.set(instance, closureInstances.get(i)); + } + + return instance; + } + + private static final class ClosureInfo { + final MALExpressionModel.ClosureArgument closure; + final String interfaceType; + int fieldIndex; + + ClosureInfo(final MALExpressionModel.ClosureArgument closure, + final String interfaceType) { + this.closure = closure; + this.interfaceType = interfaceType; + } + } + + private void collectClosures(final MALExpressionModel.Expr expr, + final List closures) { + if (expr instanceof MALExpressionModel.MetricExpr) { + collectClosuresFromChain( + ((MALExpressionModel.MetricExpr) expr).getMethodChain(), closures); + } else if (expr instanceof MALExpressionModel.BinaryExpr) { + collectClosures(((MALExpressionModel.BinaryExpr) expr).getLeft(), closures); + collectClosures(((MALExpressionModel.BinaryExpr) expr).getRight(), closures); + } else if (expr instanceof MALExpressionModel.UnaryNegExpr) { + collectClosures( + ((MALExpressionModel.UnaryNegExpr) expr).getOperand(), closures); + } else if (expr instanceof MALExpressionModel.ParenChainExpr) { + collectClosures( + ((MALExpressionModel.ParenChainExpr) expr).getInner(), closures); + collectClosuresFromChain( + ((MALExpressionModel.ParenChainExpr) expr).getMethodChain(), closures); + } else if (expr instanceof MALExpressionModel.FunctionCallExpr) { + collectClosuresFromArgs( + ((MALExpressionModel.FunctionCallExpr) expr).getArguments(), closures); + collectClosuresFromChain( + ((MALExpressionModel.FunctionCallExpr) expr).getMethodChain(), closures); + } + } + + private void collectClosuresFromChain(final List chain, + final List closures) { + for (final MALExpressionModel.MethodCall mc : chain) { + collectClosuresFromArgs(mc.getArguments(), closures); + } + } + + private void collectClosuresFromArgs(final List args, + final List closures) { + for (final MALExpressionModel.Argument arg : args) { + if (arg instanceof MALExpressionModel.ClosureArgument) { + final ClosureInfo info = new ClosureInfo( + (MALExpressionModel.ClosureArgument) arg, + "org.apache.skywalking.oap.meter.analyzer.dsl" + + ".SampleFamilyFunctions$TagFunction"); + info.fieldIndex = closures.size(); + closures.add(info); + } else if (arg instanceof MALExpressionModel.ExprArgument) { + collectClosures( + ((MALExpressionModel.ExprArgument) arg).getExpr(), closures); + } + } + } + + private Object compileClosureClass(final String className, + final ClosureInfo info) throws Exception { + final CtClass ctClass = classPool.makeClass(className); + ctClass.addInterface(classPool.get(info.interfaceType)); + ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); + + final MALExpressionModel.ClosureArgument closure = info.closure; + final List params = closure.getParams(); + final String paramName = params.isEmpty() ? "it" : params.get(0); + + final StringBuilder sb = new StringBuilder(); + sb.append("public java.util.Map apply(java.util.Map ").append(paramName) + .append(") {\n"); + for (final MALExpressionModel.ClosureStatement stmt : closure.getBody()) { + generateClosureStatement(sb, stmt, paramName); + } + sb.append(" return ").append(paramName).append(";\n"); + sb.append("}\n"); + + // Also add the Object apply(Object) bridge method + ctClass.addMethod(CtNewMethod.make(sb.toString(), ctClass)); + ctClass.addMethod(CtNewMethod.make( + "public Object apply(Object o) { return apply((java.util.Map) o); }", + ctClass)); + + final Class clazz = ctClass.toClass(MalExpressionPackageHolder.class); + ctClass.detach(); + return clazz.getDeclaredConstructor().newInstance(); + } + + private String generateRunMethod(final MALExpressionModel.Expr ast) { + final StringBuilder sb = new StringBuilder(); + sb.append("public ").append(SF).append(" run(java.util.Map samples) {\n"); + sb.append(" return "); + generateExpr(sb, ast); + sb.append(";\n"); + sb.append("}\n"); + return sb.toString(); + } + + private void generateExpr(final StringBuilder sb, + final MALExpressionModel.Expr expr) { + if (expr instanceof MALExpressionModel.MetricExpr) { + generateMetricExpr(sb, (MALExpressionModel.MetricExpr) expr); + } else if (expr instanceof MALExpressionModel.NumberExpr) { + final double val = ((MALExpressionModel.NumberExpr) expr).getValue(); + sb.append(SF).append(".EMPTY.plus(Double.valueOf(").append(val).append("))"); + } else if (expr instanceof MALExpressionModel.BinaryExpr) { + generateBinaryExpr(sb, (MALExpressionModel.BinaryExpr) expr); + } else if (expr instanceof MALExpressionModel.UnaryNegExpr) { + sb.append("("); + generateExpr(sb, ((MALExpressionModel.UnaryNegExpr) expr).getOperand()); + sb.append(").negative()"); + } else if (expr instanceof MALExpressionModel.FunctionCallExpr) { + generateFunctionCallExpr(sb, (MALExpressionModel.FunctionCallExpr) expr); + } else if (expr instanceof MALExpressionModel.ParenChainExpr) { + generateParenChainExpr(sb, (MALExpressionModel.ParenChainExpr) expr); + } + } + + private void generateMetricExpr(final StringBuilder sb, + final MALExpressionModel.MetricExpr expr) { + sb.append("((").append(SF) + .append(") samples.getOrDefault(\"") + .append(escapeJava(expr.getMetricName())) + .append("\", ").append(SF).append(".EMPTY))"); + generateMethodChain(sb, expr.getMethodChain()); + } + + private void generateFunctionCallExpr(final StringBuilder sb, + final MALExpressionModel.FunctionCallExpr expr) { + // Top-level functions like count(metric), topN(metric, n, Order) + // These are static-style calls on the first argument (SampleFamily) + final String fn = expr.getFunctionName(); + final List args = expr.getArguments(); + + if (("count".equals(fn) || "topN".equals(fn)) && !args.isEmpty()) { + // First arg is the SampleFamily + final MALExpressionModel.Argument firstArg = args.get(0); + if (firstArg instanceof MALExpressionModel.ExprArgument) { + generateExpr(sb, + ((MALExpressionModel.ExprArgument) firstArg).getExpr()); + } + sb.append('.').append(fn).append('('); + for (int i = 1; i < args.size(); i++) { + if (i > 1) { + sb.append(", "); + } + generateArgument(sb, args.get(i)); + } + sb.append(')'); + } else { + // Generic function call + sb.append(fn).append('('); + for (int i = 0; i < args.size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateArgument(sb, args.get(i)); + } + sb.append(')'); + } + generateMethodChain(sb, expr.getMethodChain()); + } + + private void generateParenChainExpr(final StringBuilder sb, + final MALExpressionModel.ParenChainExpr expr) { + sb.append("("); + generateExpr(sb, expr.getInner()); + sb.append(")"); + generateMethodChain(sb, expr.getMethodChain()); + } + + private void generateBinaryExpr(final StringBuilder sb, + final MALExpressionModel.BinaryExpr expr) { + final MALExpressionModel.Expr left = expr.getLeft(); + final MALExpressionModel.Expr right = expr.getRight(); + final MALExpressionModel.ArithmeticOp op = expr.getOp(); + + final boolean leftIsNumber = left instanceof MALExpressionModel.NumberExpr; + final boolean rightIsNumber = right instanceof MALExpressionModel.NumberExpr; + + if (leftIsNumber && !rightIsNumber) { + // N op SF -> swap to SF.op(N) with special handling for SUB and DIV + final double num = ((MALExpressionModel.NumberExpr) left).getValue(); + switch (op) { + case ADD: + sb.append("("); + generateExpr(sb, right); + sb.append(").plus(Double.valueOf(").append(num).append("))"); + break; + case SUB: + sb.append("("); + generateExpr(sb, right); + sb.append(").minus(Double.valueOf(") + .append(num).append(")).negative()"); + break; + case MUL: + sb.append("("); + generateExpr(sb, right); + sb.append(").multiply(Double.valueOf(").append(num).append("))"); + break; + case DIV: + sb.append("("); + generateExpr(sb, right); + sb.append(").newValue(new java.util.function.Function() { ") + .append("public Object apply(Object v) { ") + .append("return Double.valueOf(").append(num) + .append(" / ((Double)v).doubleValue()); } })"); + break; + default: + throw new IllegalArgumentException("Unsupported op: " + op); + } + } else if (!leftIsNumber && rightIsNumber) { + // SF op N + final double num = ((MALExpressionModel.NumberExpr) right).getValue(); + sb.append("("); + generateExpr(sb, left); + sb.append(").").append(opMethodName(op)) + .append("(Double.valueOf(").append(num).append("))"); + } else { + // SF op SF (both non-number) + sb.append("("); + generateExpr(sb, left); + sb.append(").").append(opMethodName(op)).append("("); + generateExpr(sb, right); + sb.append(")"); + } + } + + /** + * Methods on SampleFamily that take String[] (varargs). + * Javassist doesn't support varargs syntax, so multiple string args + * must be wrapped in {@code new String[]{}}. + */ + private static final Set VARARGS_STRING_METHODS = Set.of( + "tagEqual", "tagNotEqual", "tagMatch", "tagNotMatch" + ); + + private void generateMethodChain(final StringBuilder sb, + final List chain) { + for (final MALExpressionModel.MethodCall mc : chain) { + sb.append('.').append(mc.getName()).append('('); + final List args = mc.getArguments(); + if (VARARGS_STRING_METHODS.contains(mc.getName()) && !args.isEmpty() + && allStringArgs(args)) { + sb.append("new String[]{"); + for (int i = 0; i < args.size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateArgument(sb, args.get(i)); + } + sb.append('}'); + } else { + for (int i = 0; i < args.size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateArgument(sb, args.get(i)); + } + } + sb.append(')'); + } + } + + private static boolean allStringArgs(final List args) { + for (final MALExpressionModel.Argument arg : args) { + if (!(arg instanceof MALExpressionModel.StringArgument)) { + return false; + } + } + return true; + } + + private void generateArgument(final StringBuilder sb, + final MALExpressionModel.Argument arg) { + if (arg instanceof MALExpressionModel.StringArgument) { + sb.append('"') + .append(escapeJava(((MALExpressionModel.StringArgument) arg).getValue())) + .append('"'); + } else if (arg instanceof MALExpressionModel.StringListArgument) { + final List vals = + ((MALExpressionModel.StringListArgument) arg).getValues(); + sb.append("java.util.List.of("); + for (int i = 0; i < vals.size(); i++) { + if (i > 0) { + sb.append(", "); + } + sb.append('"').append(escapeJava(vals.get(i))).append('"'); + } + sb.append(')'); + } else if (arg instanceof MALExpressionModel.NumberListArgument) { + final List vals = + ((MALExpressionModel.NumberListArgument) arg).getValues(); + sb.append("java.util.List.of("); + for (int i = 0; i < vals.size(); i++) { + if (i > 0) { + sb.append(", "); + } + final double v = vals.get(i); + if (v == Math.floor(v) && !Double.isInfinite(v)) { + sb.append("Integer.valueOf(").append((int) v).append(')'); + } else { + sb.append("Double.valueOf(").append(v).append(')'); + } + } + sb.append(')'); + } else if (arg instanceof MALExpressionModel.BoolArgument) { + sb.append(((MALExpressionModel.BoolArgument) arg).isValue()); + } else if (arg instanceof MALExpressionModel.EnumRefArgument) { + final MALExpressionModel.EnumRefArgument enumRef = + (MALExpressionModel.EnumRefArgument) arg; + final String fqcn = ENUM_FQCN.get(enumRef.getEnumType()); + if (fqcn != null) { + sb.append(fqcn); + } else { + sb.append(enumRef.getEnumType()); + } + sb.append('.').append(enumRef.getEnumValue()); + } else if (arg instanceof MALExpressionModel.ExprArgument) { + final MALExpressionModel.Expr innerExpr = + ((MALExpressionModel.ExprArgument) arg).getExpr(); + if (innerExpr instanceof MALExpressionModel.MetricExpr + && ((MALExpressionModel.MetricExpr) innerExpr).getMethodChain().isEmpty()) { + // Bare identifier — could be an enum constant like SUM, AVG + final String name = + ((MALExpressionModel.MetricExpr) innerExpr).getMetricName(); + if (isDownsamplingType(name)) { + sb.append(ENUM_FQCN.get("DownsamplingType")).append('.').append(name); + } else { + // It's a metric reference used as argument (e.g., div(other_metric)) + generateExpr(sb, innerExpr); + } + } else { + generateExpr(sb, innerExpr); + } + } else if (arg instanceof MALExpressionModel.ClosureArgument) { + generateClosureArgument(sb, (MALExpressionModel.ClosureArgument) arg); + } + } + + private void generateClosureArgument(final StringBuilder sb, + final MALExpressionModel.ClosureArgument closure) { + // Reference pre-compiled closure field + sb.append("this._closure").append(closureCounter++); + } + + private void generateClosureStatement(final StringBuilder sb, + final MALExpressionModel.ClosureStatement stmt, + final String paramName) { + if (stmt instanceof MALExpressionModel.ClosureAssignment) { + final MALExpressionModel.ClosureAssignment assign = + (MALExpressionModel.ClosureAssignment) stmt; + sb.append(" ").append(paramName).append(".put(\"") + .append(escapeJava(assign.getTarget())).append("\", "); + generateClosureExpr(sb, assign.getValue(), paramName); + sb.append(");\n"); + } else if (stmt instanceof MALExpressionModel.ClosureIfStatement) { + final MALExpressionModel.ClosureIfStatement ifStmt = + (MALExpressionModel.ClosureIfStatement) stmt; + sb.append(" if ("); + generateClosureCondition(sb, ifStmt.getCondition(), paramName); + sb.append(") {\n"); + for (final MALExpressionModel.ClosureStatement s : ifStmt.getThenBranch()) { + generateClosureStatement(sb, s, paramName); + } + sb.append(" }\n"); + if (!ifStmt.getElseBranch().isEmpty()) { + sb.append(" else {\n"); + for (final MALExpressionModel.ClosureStatement s : ifStmt.getElseBranch()) { + generateClosureStatement(sb, s, paramName); + } + sb.append(" }\n"); + } + } else if (stmt instanceof MALExpressionModel.ClosureReturnStatement) { + sb.append(" return (java.util.Map) "); + generateClosureExpr(sb, + ((MALExpressionModel.ClosureReturnStatement) stmt).getValue(), paramName); + sb.append(";\n"); + } else if (stmt instanceof MALExpressionModel.ClosureExprStatement) { + sb.append(" "); + generateClosureExpr(sb, + ((MALExpressionModel.ClosureExprStatement) stmt).getExpr(), paramName); + sb.append(";\n"); + } + } + + private void generateClosureExpr(final StringBuilder sb, + final MALExpressionModel.ClosureExpr expr, + final String paramName) { + if (expr instanceof MALExpressionModel.ClosureStringLiteral) { + sb.append('"') + .append(escapeJava(((MALExpressionModel.ClosureStringLiteral) expr).getValue())) + .append('"'); + } else if (expr instanceof MALExpressionModel.ClosureNumberLiteral) { + sb.append(((MALExpressionModel.ClosureNumberLiteral) expr).getValue()); + } else if (expr instanceof MALExpressionModel.ClosureBoolLiteral) { + sb.append(((MALExpressionModel.ClosureBoolLiteral) expr).isValue()); + } else if (expr instanceof MALExpressionModel.ClosureNullLiteral) { + sb.append("null"); + } else if (expr instanceof MALExpressionModel.ClosureMethodChain) { + generateClosureMethodChain(sb, + (MALExpressionModel.ClosureMethodChain) expr, paramName); + } else if (expr instanceof MALExpressionModel.ClosureBinaryExpr) { + final MALExpressionModel.ClosureBinaryExpr bin = + (MALExpressionModel.ClosureBinaryExpr) expr; + sb.append("("); + generateClosureExpr(sb, bin.getLeft(), paramName); + switch (bin.getOp()) { + case ADD: + sb.append(" + "); + break; + case SUB: + sb.append(" - "); + break; + case MUL: + sb.append(" * "); + break; + case DIV: + sb.append(" / "); + break; + default: + break; + } + generateClosureExpr(sb, bin.getRight(), paramName); + sb.append(")"); + } + } + + private void generateClosureMethodChain( + final StringBuilder sb, + final MALExpressionModel.ClosureMethodChain chain, + final String paramName) { + // tags.key -> tags.get("key") + // tags['key'] -> tags.get("key") + final List segs = chain.getSegments(); + if (segs.size() == 1 + && segs.get(0) instanceof MALExpressionModel.ClosureFieldAccess) { + final String key = + ((MALExpressionModel.ClosureFieldAccess) segs.get(0)).getName(); + sb.append(chain.getTarget()).append(".get(\"") + .append(escapeJava(key)).append("\")"); + } else if (segs.size() == 1 + && segs.get(0) instanceof MALExpressionModel.ClosureIndexAccess) { + sb.append(chain.getTarget()).append(".get("); + generateClosureExpr(sb, + ((MALExpressionModel.ClosureIndexAccess) segs.get(0)).getIndex(), paramName); + sb.append(")"); + } else { + // General chain + sb.append(chain.getTarget()); + for (final MALExpressionModel.ClosureChainSegment seg : segs) { + if (seg instanceof MALExpressionModel.ClosureFieldAccess) { + sb.append(".get(\"") + .append(escapeJava( + ((MALExpressionModel.ClosureFieldAccess) seg).getName())) + .append("\")"); + } else if (seg instanceof MALExpressionModel.ClosureIndexAccess) { + sb.append(".get("); + generateClosureExpr(sb, + ((MALExpressionModel.ClosureIndexAccess) seg).getIndex(), paramName); + sb.append(")"); + } else if (seg instanceof MALExpressionModel.ClosureMethodCallSeg) { + final MALExpressionModel.ClosureMethodCallSeg mc = + (MALExpressionModel.ClosureMethodCallSeg) seg; + sb.append('.').append(mc.getName()).append('('); + for (int i = 0; i < mc.getArguments().size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateClosureExpr(sb, mc.getArguments().get(i), paramName); + } + sb.append(')'); + } + } + } + } + + private void generateClosureCondition(final StringBuilder sb, + final MALExpressionModel.ClosureCondition cond, + final String paramName) { + if (cond instanceof MALExpressionModel.ClosureComparison) { + final MALExpressionModel.ClosureComparison cc = + (MALExpressionModel.ClosureComparison) cond; + switch (cc.getOp()) { + case EQ: + sb.append("java.util.Objects.equals("); + generateClosureExpr(sb, cc.getLeft(), paramName); + sb.append(", "); + generateClosureExpr(sb, cc.getRight(), paramName); + sb.append(")"); + break; + case NEQ: + sb.append("!java.util.Objects.equals("); + generateClosureExpr(sb, cc.getLeft(), paramName); + sb.append(", "); + generateClosureExpr(sb, cc.getRight(), paramName); + sb.append(")"); + break; + default: + generateClosureExpr(sb, cc.getLeft(), paramName); + sb.append(comparisonOperator(cc.getOp())); + generateClosureExpr(sb, cc.getRight(), paramName); + break; + } + } else if (cond instanceof MALExpressionModel.ClosureLogical) { + final MALExpressionModel.ClosureLogical lc = + (MALExpressionModel.ClosureLogical) cond; + sb.append("("); + generateClosureCondition(sb, lc.getLeft(), paramName); + sb.append(lc.getOp() == MALExpressionModel.LogicalOp.AND ? " && " : " || "); + generateClosureCondition(sb, lc.getRight(), paramName); + sb.append(")"); + } else if (cond instanceof MALExpressionModel.ClosureNot) { + sb.append("!("); + generateClosureCondition(sb, + ((MALExpressionModel.ClosureNot) cond).getInner(), paramName); + sb.append(")"); + } else if (cond instanceof MALExpressionModel.ClosureExprCondition) { + // Truthiness check + sb.append("("); + generateClosureExpr(sb, + ((MALExpressionModel.ClosureExprCondition) cond).getExpr(), paramName); + sb.append(" != null)"); + } else if (cond instanceof MALExpressionModel.ClosureInCondition) { + final MALExpressionModel.ClosureInCondition ic = + (MALExpressionModel.ClosureInCondition) cond; + sb.append("java.util.List.of("); + for (int i = 0; i < ic.getValues().size(); i++) { + if (i > 0) { + sb.append(", "); + } + sb.append('"').append(escapeJava(ic.getValues().get(i))).append('"'); + } + sb.append(").contains("); + generateClosureExpr(sb, ic.getExpr(), paramName); + sb.append(")"); + } + } + + private static void collectSampleNames(final MALExpressionModel.Expr expr, + final Set names) { + if (expr instanceof MALExpressionModel.MetricExpr) { + names.add(((MALExpressionModel.MetricExpr) expr).getMetricName()); + } else if (expr instanceof MALExpressionModel.BinaryExpr) { + collectSampleNames(((MALExpressionModel.BinaryExpr) expr).getLeft(), names); + collectSampleNames(((MALExpressionModel.BinaryExpr) expr).getRight(), names); + } else if (expr instanceof MALExpressionModel.UnaryNegExpr) { + collectSampleNames( + ((MALExpressionModel.UnaryNegExpr) expr).getOperand(), names); + } else if (expr instanceof MALExpressionModel.ParenChainExpr) { + collectSampleNames( + ((MALExpressionModel.ParenChainExpr) expr).getInner(), names); + } else if (expr instanceof MALExpressionModel.FunctionCallExpr) { + for (final MALExpressionModel.Argument arg : + ((MALExpressionModel.FunctionCallExpr) expr).getArguments()) { + if (arg instanceof MALExpressionModel.ExprArgument) { + collectSampleNames( + ((MALExpressionModel.ExprArgument) arg).getExpr(), names); + } + } + } + } + + /** + * Extracts compile-time metadata from the AST by walking all method chains. + */ + static ExpressionMetadata extractMetadata(final MALExpressionModel.Expr ast) { + final Set sampleNames = new HashSet<>(); + collectSampleNames(ast, sampleNames); + + ScopeType scopeType = null; + final Set scopeLabels = new LinkedHashSet<>(); + final Set aggregationLabels = new LinkedHashSet<>(); + DownsamplingType downsampling = DownsamplingType.AVG; + boolean isHistogram = false; + int[] percentiles = null; + + final List> allChains = new ArrayList<>(); + collectMethodChains(ast, allChains); + + for (final List chain : allChains) { + for (final MALExpressionModel.MethodCall mc : chain) { + final String name = mc.getName(); + switch (name) { + case "sum": + case "avg": + case "max": + case "min": + addStringListLabels(mc, aggregationLabels); + break; + case "count": + addStringListLabels(mc, aggregationLabels); + break; + case "service": + scopeType = ScopeType.SERVICE; + addStringListLabels(mc, scopeLabels); + break; + case "instance": + scopeType = ScopeType.SERVICE_INSTANCE; + addAllStringListLabels(mc, scopeLabels); + break; + case "endpoint": + scopeType = ScopeType.ENDPOINT; + addAllStringListLabels(mc, scopeLabels); + break; + case "process": + scopeType = ScopeType.PROCESS; + addAllStringListLabels(mc, scopeLabels); + addStringArgLabels(mc, scopeLabels); + break; + case "serviceRelation": + scopeType = ScopeType.SERVICE_RELATION; + addAllStringListLabels(mc, scopeLabels); + addStringArgLabels(mc, scopeLabels); + break; + case "processRelation": + scopeType = ScopeType.PROCESS_RELATION; + addAllStringListLabels(mc, scopeLabels); + addStringArgLabels(mc, scopeLabels); + break; + case "histogram": + isHistogram = true; + break; + case "histogram_percentile": + if (!mc.getArguments().isEmpty() + && mc.getArguments().get(0) instanceof MALExpressionModel.NumberListArgument) { + final List vals = + ((MALExpressionModel.NumberListArgument) mc.getArguments().get(0)).getValues(); + percentiles = new int[vals.size()]; + for (int i = 0; i < vals.size(); i++) { + percentiles[i] = vals.get(i).intValue(); + } + } + break; + case "downsampling": + if (!mc.getArguments().isEmpty() + && mc.getArguments().get(0) instanceof MALExpressionModel.EnumRefArgument) { + final String val = + ((MALExpressionModel.EnumRefArgument) mc.getArguments().get(0)).getEnumValue(); + downsampling = DownsamplingType.valueOf(val); + } + break; + default: + break; + } + } + } + + return new ExpressionMetadata( + new ArrayList<>(sampleNames), + scopeType, + scopeLabels, + aggregationLabels, + downsampling, + isHistogram, + percentiles + ); + } + + private static void addStringListLabels(final MALExpressionModel.MethodCall mc, + final Set target) { + if (!mc.getArguments().isEmpty() + && mc.getArguments().get(0) instanceof MALExpressionModel.StringListArgument) { + target.addAll( + ((MALExpressionModel.StringListArgument) mc.getArguments().get(0)).getValues()); + } + } + + private static void addAllStringListLabels(final MALExpressionModel.MethodCall mc, + final Set target) { + for (final MALExpressionModel.Argument arg : mc.getArguments()) { + if (arg instanceof MALExpressionModel.StringListArgument) { + target.addAll(((MALExpressionModel.StringListArgument) arg).getValues()); + } + } + } + + private static void addStringArgLabels(final MALExpressionModel.MethodCall mc, + final Set target) { + for (final MALExpressionModel.Argument arg : mc.getArguments()) { + if (arg instanceof MALExpressionModel.StringArgument) { + target.add(((MALExpressionModel.StringArgument) arg).getValue()); + } + } + } + + private static void collectMethodChains(final MALExpressionModel.Expr expr, + final List> chains) { + if (expr instanceof MALExpressionModel.MetricExpr) { + chains.add(((MALExpressionModel.MetricExpr) expr).getMethodChain()); + } else if (expr instanceof MALExpressionModel.BinaryExpr) { + collectMethodChains(((MALExpressionModel.BinaryExpr) expr).getLeft(), chains); + collectMethodChains(((MALExpressionModel.BinaryExpr) expr).getRight(), chains); + } else if (expr instanceof MALExpressionModel.UnaryNegExpr) { + collectMethodChains(((MALExpressionModel.UnaryNegExpr) expr).getOperand(), chains); + } else if (expr instanceof MALExpressionModel.ParenChainExpr) { + collectMethodChains(((MALExpressionModel.ParenChainExpr) expr).getInner(), chains); + chains.add(((MALExpressionModel.ParenChainExpr) expr).getMethodChain()); + } else if (expr instanceof MALExpressionModel.FunctionCallExpr) { + for (final MALExpressionModel.Argument arg : + ((MALExpressionModel.FunctionCallExpr) expr).getArguments()) { + if (arg instanceof MALExpressionModel.ExprArgument) { + collectMethodChains(((MALExpressionModel.ExprArgument) arg).getExpr(), chains); + } + } + chains.add(((MALExpressionModel.FunctionCallExpr) expr).getMethodChain()); + } + } + + private String generateMetadataMethod(final ExpressionMetadata metadata) { + final StringBuilder sb = new StringBuilder(); + final String mdClass = "org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionMetadata"; + final String scopeTypeClass = "org.apache.skywalking.oap.server.core.analysis.meter.ScopeType"; + final String dsTypeClass = "org.apache.skywalking.oap.meter.analyzer.dsl.DownsamplingType"; + + sb.append("public ").append(mdClass).append(" metadata() {\n"); + + // samples list + sb.append(" java.util.List _samples = new java.util.ArrayList();\n"); + for (final String sample : metadata.getSamples()) { + sb.append(" _samples.add(\"").append(escapeJava(sample)).append("\");\n"); + } + + // scope labels set + sb.append(" java.util.Set _scopeLabels = new java.util.LinkedHashSet();\n"); + for (final String label : metadata.getScopeLabels()) { + sb.append(" _scopeLabels.add(\"").append(escapeJava(label)).append("\");\n"); + } + + // aggregation labels set + sb.append(" java.util.Set _aggLabels = new java.util.LinkedHashSet();\n"); + for (final String label : metadata.getAggregationLabels()) { + sb.append(" _aggLabels.add(\"").append(escapeJava(label)).append("\");\n"); + } + + // percentiles array + if (metadata.getPercentiles() != null) { + sb.append(" int[] _pct = new int[]{"); + for (int i = 0; i < metadata.getPercentiles().length; i++) { + if (i > 0) { + sb.append(", "); + } + sb.append(metadata.getPercentiles()[i]); + } + sb.append("};\n"); + } else { + sb.append(" int[] _pct = null;\n"); + } + + sb.append(" return new ").append(mdClass).append("(\n"); + sb.append(" _samples,\n"); + if (metadata.getScopeType() != null) { + sb.append(" ").append(scopeTypeClass).append('.').append(metadata.getScopeType().name()).append(",\n"); + } else { + sb.append(" null,\n"); + } + sb.append(" _scopeLabels,\n"); + sb.append(" _aggLabels,\n"); + sb.append(" ").append(dsTypeClass).append('.').append(metadata.getDownsampling().name()).append(",\n"); + sb.append(" ").append(metadata.isHistogram()).append(",\n"); + sb.append(" _pct\n"); + sb.append(" );\n"); + sb.append("}\n"); + return sb.toString(); + } + + private static String opMethodName(final MALExpressionModel.ArithmeticOp op) { + switch (op) { + case ADD: + return "plus"; + case SUB: + return "minus"; + case MUL: + return "multiply"; + case DIV: + return "div"; + default: + throw new IllegalArgumentException("Unknown op: " + op); + } + } + + private static String comparisonOperator(final MALExpressionModel.CompareOp op) { + switch (op) { + case GT: + return " > "; + case LT: + return " < "; + case GTE: + return " >= "; + case LTE: + return " <= "; + default: + return " == "; + } + } + + private static boolean isDownsamplingType(final String name) { + return "AVG".equals(name) || "SUM".equals(name) || "LATEST".equals(name) + || "SUM_PER_MIN".equals(name) || "MAX".equals(name) || "MIN".equals(name); + } + + private static String escapeJava(final String s) { + return s.replace("\\", "\\\\") + .replace("\"", "\\\"") + .replace("\n", "\\n") + .replace("\r", "\\r") + .replace("\t", "\\t"); + } + + /** + * Generates the Java source body of the run method for debugging/testing. + */ + public String generateSource(final String expression) { + closureCounter = 0; + final MALExpressionModel.Expr ast = MALScriptParser.parse(expression); + return generateRunMethod(ast); + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java new file mode 100644 index 000000000000..3ba103ed839b --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java @@ -0,0 +1,480 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.compiler; + +import java.util.Collections; +import java.util.List; +import lombok.Getter; + +/** + * Immutable AST model for MAL (Meter Analysis Language) expressions. + * + *

Represents parsed expressions like: + * 

+ *   metric_name.tagEqual("k","v").sum(["tag"]).rate("PT1M").service(["svc"], Layer.GENERAL)
+ *   (metric1 + metric2) * 100
+ *   metric.tag({tags -> tags.key = "val"}).histogram().histogram_percentile([50,75,90,95,99])
+ *
+ */ +public final class MALExpressionModel { + + // ==================== Expression nodes ==================== + + /** + * Base interface for all expression AST nodes. + */ + public interface Expr { + } + + /** + * Metric reference with optional method chain: + * {@code metric_name} or {@code metric_name.sum(["tag"]).rate("PT1M")} + */ + @Getter + public static final class MetricExpr implements Expr { + private final String metricName; + private final List methodChain; + + public MetricExpr(final String metricName, final List methodChain) { + this.metricName = metricName; + this.methodChain = Collections.unmodifiableList(methodChain); + } + } + + /** + * Top-level function call: {@code count(metric)}, {@code topN(metric, 10, Order.ASC)} + */ + @Getter + public static final class FunctionCallExpr implements Expr { + private final String functionName; + private final List arguments; + private final List methodChain; + + public FunctionCallExpr(final String functionName, + final List arguments, + final List methodChain) { + this.functionName = functionName; + this.arguments = Collections.unmodifiableList(arguments); + this.methodChain = Collections.unmodifiableList(methodChain); + } + } + + /** + * Parenthesized expression with method chain: + * {@code (metric * 100).sum(['tag']).rate('PT1M')} + */ + @Getter + public static final class ParenChainExpr implements Expr { + private final Expr inner; + private final List methodChain; + + public ParenChainExpr(final Expr inner, final List methodChain) { + this.inner = inner; + this.methodChain = Collections.unmodifiableList(methodChain); + } + } + + /** + * Binary arithmetic: {@code metric1 + metric2}, {@code (metric * 100)} + */ + @Getter + public static final class BinaryExpr implements Expr { + private final Expr left; + private final ArithmeticOp op; + private final Expr right; + + public BinaryExpr(final Expr left, final ArithmeticOp op, final Expr right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + /** + * Unary negation: {@code -metric} + */ + @Getter + public static final class UnaryNegExpr implements Expr { + private final Expr operand; + + public UnaryNegExpr(final Expr operand) { + this.operand = operand; + } + } + + /** + * Numeric literal: {@code 100}, {@code 3.14} + */ + @Getter + public static final class NumberExpr implements Expr { + private final double value; + + public NumberExpr(final double value) { + this.value = value; + } + } + + // ==================== Method calls ==================== + + /** + * A method call in a chain: {@code .sum(["tag"])}, {@code .rate("PT1M")} + */ + @Getter + public static final class MethodCall { + private final String name; + private final List arguments; + + public MethodCall(final String name, final List arguments) { + this.name = name; + this.arguments = Collections.unmodifiableList(arguments); + } + } + + // ==================== Arguments ==================== + + /** + * Base interface for method/function arguments. + */ + public interface Argument { + } + + /** + * Expression argument (metric ref, number, arithmetic). + */ + @Getter + public static final class ExprArgument implements Argument { + private final Expr expr; + + public ExprArgument(final Expr expr) { + this.expr = expr; + } + } + + /** + * String list: {@code ["tag1", "tag2"]} + */ + @Getter + public static final class StringListArgument implements Argument { + private final List values; + + public StringListArgument(final List values) { + this.values = Collections.unmodifiableList(values); + } + } + + /** + * Number list: {@code [50, 75, 90, 95, 99]} + */ + @Getter + public static final class NumberListArgument implements Argument { + private final List values; + + public NumberListArgument(final List values) { + this.values = Collections.unmodifiableList(values); + } + } + + /** + * String literal: {@code "PT1M"}, {@code 'command'} + */ + @Getter + public static final class StringArgument implements Argument { + private final String value; + + public StringArgument(final String value) { + this.value = value; + } + } + + /** + * Boolean literal: {@code true}, {@code false} + */ + @Getter + public static final class BoolArgument implements Argument { + private final boolean value; + + public BoolArgument(final boolean value) { + this.value = value; + } + } + + /** + * Enum reference: {@code Layer.GENERAL}, {@code K8sRetagType.Pod2Service} + */ + @Getter + public static final class EnumRefArgument implements Argument { + private final String enumType; + private final String enumValue; + + public EnumRefArgument(final String enumType, final String enumValue) { + this.enumType = enumType; + this.enumValue = enumValue; + } + } + + /** + * Closure expression: {@code {tags -> tags.key = "val"}} + */ + @Getter + public static final class ClosureArgument implements Argument { + private final List params; + private final List body; + + public ClosureArgument(final List params, final List body) { + this.params = Collections.unmodifiableList(params); + this.body = Collections.unmodifiableList(body); + } + } + + // ==================== Closure statements ==================== + + public interface ClosureStatement { + } + + @Getter + public static final class ClosureIfStatement implements ClosureStatement { + private final ClosureCondition condition; + private final List thenBranch; + private final List elseBranch; + + public ClosureIfStatement(final ClosureCondition condition, + final List thenBranch, + final List elseBranch) { + this.condition = condition; + this.thenBranch = Collections.unmodifiableList(thenBranch); + this.elseBranch = elseBranch != null + ? Collections.unmodifiableList(elseBranch) : Collections.emptyList(); + } + } + + @Getter + public static final class ClosureReturnStatement implements ClosureStatement { + private final ClosureExpr value; + + public ClosureReturnStatement(final ClosureExpr value) { + this.value = value; + } + } + + @Getter + public static final class ClosureAssignment implements ClosureStatement { + private final String target; + private final ClosureExpr value; + + public ClosureAssignment(final String target, final ClosureExpr value) { + this.target = target; + this.value = value; + } + } + + @Getter + public static final class ClosureExprStatement implements ClosureStatement { + private final ClosureExpr expr; + + public ClosureExprStatement(final ClosureExpr expr) { + this.expr = expr; + } + } + + // ==================== Closure expressions ==================== + + public interface ClosureExpr { + } + + @Getter + public static final class ClosureStringLiteral implements ClosureExpr { + private final String value; + + public ClosureStringLiteral(final String value) { + this.value = value; + } + } + + @Getter + public static final class ClosureNumberLiteral implements ClosureExpr { + private final double value; + + public ClosureNumberLiteral(final double value) { + this.value = value; + } + } + + @Getter + public static final class ClosureBoolLiteral implements ClosureExpr { + private final boolean value; + + public ClosureBoolLiteral(final boolean value) { + this.value = value; + } + } + + public static final class ClosureNullLiteral implements ClosureExpr { + } + + /** + * Method chain in closure: {@code tags.service_name}, {@code tags['key']}, + * {@code tags.service?.trim()} + */ + @Getter + public static final class ClosureMethodChain implements ClosureExpr { + private final String target; + private final List segments; + + public ClosureMethodChain(final String target, + final List segments) { + this.target = target; + this.segments = Collections.unmodifiableList(segments); + } + } + + @Getter + public static final class ClosureBinaryExpr implements ClosureExpr { + private final ClosureExpr left; + private final ArithmeticOp op; + private final ClosureExpr right; + + public ClosureBinaryExpr(final ClosureExpr left, + final ArithmeticOp op, + final ClosureExpr right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + // ==================== Closure chain segments ==================== + + public interface ClosureChainSegment { + } + + @Getter + public static final class ClosureFieldAccess implements ClosureChainSegment { + private final String name; + private final boolean safeNav; + + public ClosureFieldAccess(final String name, final boolean safeNav) { + this.name = name; + this.safeNav = safeNav; + } + } + + @Getter + public static final class ClosureMethodCallSeg implements ClosureChainSegment { + private final String name; + private final List arguments; + private final boolean safeNav; + + public ClosureMethodCallSeg(final String name, + final List arguments, + final boolean safeNav) { + this.name = name; + this.arguments = Collections.unmodifiableList(arguments); + this.safeNav = safeNav; + } + } + + @Getter + public static final class ClosureIndexAccess implements ClosureChainSegment { + private final ClosureExpr index; + + public ClosureIndexAccess(final ClosureExpr index) { + this.index = index; + } + } + + // ==================== Closure conditions ==================== + + public interface ClosureCondition { + } + + @Getter + public static final class ClosureComparison implements ClosureCondition { + private final ClosureExpr left; + private final CompareOp op; + private final ClosureExpr right; + + public ClosureComparison(final ClosureExpr left, + final CompareOp op, + final ClosureExpr right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + @Getter + public static final class ClosureLogical implements ClosureCondition { + private final ClosureCondition left; + private final LogicalOp op; + private final ClosureCondition right; + + public ClosureLogical(final ClosureCondition left, + final LogicalOp op, + final ClosureCondition right) { + this.left = left; + this.op = op; + this.right = right; + } + } + + @Getter + public static final class ClosureNot implements ClosureCondition { + private final ClosureCondition inner; + + public ClosureNot(final ClosureCondition inner) { + this.inner = inner; + } + } + + @Getter + public static final class ClosureExprCondition implements ClosureCondition { + private final ClosureExpr expr; + + public ClosureExprCondition(final ClosureExpr expr) { + this.expr = expr; + } + } + + @Getter + public static final class ClosureInCondition implements ClosureCondition { + private final ClosureExpr expr; + private final List values; + + public ClosureInCondition(final ClosureExpr expr, final List values) { + this.expr = expr; + this.values = Collections.unmodifiableList(values); + } + } + + // ==================== Enums ==================== + + public enum ArithmeticOp { + ADD, SUB, MUL, DIV + } + + public enum CompareOp { + EQ, NEQ, GT, LT, GTE, LTE + } + + public enum LogicalOp { + AND, OR + } + + private MALExpressionModel() { + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java new file mode 100644 index 000000000000..ed8c7e04fbfb --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java @@ -0,0 +1,491 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.compiler; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import org.antlr.v4.runtime.BaseErrorListener; +import org.antlr.v4.runtime.CharStreams; +import org.antlr.v4.runtime.CommonTokenStream; +import org.antlr.v4.runtime.RecognitionException; +import org.antlr.v4.runtime.Recognizer; +import org.apache.skywalking.mal.rt.grammar.MALLexer; +import org.apache.skywalking.mal.rt.grammar.MALParser; +import org.apache.skywalking.mal.rt.grammar.MALParserBaseVisitor; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.Argument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ArithmeticOp; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.BinaryExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.BoolArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureAssignment; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureBinaryExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureBoolLiteral; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureChainSegment; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureCondition; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureExprCondition; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureExprStatement; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureFieldAccess; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureIfStatement; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureIndexAccess; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureMethodCallSeg; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureMethodChain; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureNullLiteral; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureNumberLiteral; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureReturnStatement; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureStatement; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureStringLiteral; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.CompareOp; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.EnumRefArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.Expr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ExprArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.FunctionCallExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.LogicalOp; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.MetricExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.MethodCall; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.NumberExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.NumberListArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ParenChainExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.StringArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.StringListArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.UnaryNegExpr; + +/** + * Facade: parses MAL expression strings into {@link MALExpressionModel.Expr} AST. + * + * 
+ *   MALExpressionModel.Expr ast = MALScriptParser.parse(
+ *       "metric.sum(['tag']).rate('PT1M').service(['svc'], Layer.GENERAL)");
+ *
+ */ +public final class MALScriptParser { + + private MALScriptParser() { + } + + public static Expr parse(final String expression) { + final MALLexer lexer = new MALLexer(CharStreams.fromString(expression)); + final CommonTokenStream tokens = new CommonTokenStream(lexer); + final MALParser parser = new MALParser(tokens); + + final List errors = new ArrayList<>(); + parser.removeErrorListeners(); + parser.addErrorListener(new BaseErrorListener() { + @Override + public void syntaxError(final Recognizer recognizer, + final Object offendingSymbol, + final int line, + final int charPositionInLine, + final String msg, + final RecognitionException e) { + errors.add(line + ":" + charPositionInLine + " " + msg); + } + }); + + final MALParser.ExpressionContext tree = parser.expression(); + if (!errors.isEmpty()) { + throw new IllegalArgumentException( + "MAL expression parsing failed: " + String.join("; ", errors) + + " in expression: " + expression); + } + + return new MALExprVisitor().visit(tree.additiveExpression()); + } + + /** + * Visitor transforming ANTLR4 parse tree into MAL expression AST. + */ + private static final class MALExprVisitor extends MALParserBaseVisitor { + + @Override + public Expr visitAdditiveExpression(final MALParser.AdditiveExpressionContext ctx) { + Expr result = visit(ctx.multiplicativeExpression(0)); + for (int i = 1; i < ctx.multiplicativeExpression().size(); i++) { + final ArithmeticOp op = ctx.getChild(2 * i - 1).getText().equals("+") + ? ArithmeticOp.ADD : ArithmeticOp.SUB; + result = new BinaryExpr(result, op, visit(ctx.multiplicativeExpression(i))); + } + return result; + } + + @Override + public Expr visitMultiplicativeExpression( + final MALParser.MultiplicativeExpressionContext ctx) { + Expr result = visit(ctx.unaryExpression(0)); + for (int i = 1; i < ctx.unaryExpression().size(); i++) { + final ArithmeticOp op = ctx.getChild(2 * i - 1).getText().equals("*") + ? ArithmeticOp.MUL : ArithmeticOp.DIV; + result = new BinaryExpr(result, op, visit(ctx.unaryExpression(i))); + } + return result; + } + + @Override + public Expr visitUnaryNeg(final MALParser.UnaryNegContext ctx) { + return new UnaryNegExpr(visit(ctx.unaryExpression())); + } + + @Override + public Expr visitUnaryPostfix(final MALParser.UnaryPostfixContext ctx) { + return visit(ctx.postfixExpression()); + } + + @Override + public Expr visitUnaryNumber(final MALParser.UnaryNumberContext ctx) { + return new NumberExpr(Double.parseDouble(ctx.NUMBER().getText())); + } + + @Override + public Expr visitPostfixExpression(final MALParser.PostfixExpressionContext ctx) { + final List methods = new ArrayList<>(); + for (final MALParser.MethodCallContext mc : ctx.methodCall()) { + methods.add(visitMethodCallNode(mc)); + } + + final MALParser.PrimaryContext primary = ctx.primary(); + if (primary.functionCall() != null) { + final MALParser.FunctionCallContext fc = primary.functionCall(); + final List args = fc.argumentList() != null + ? visitArgList(fc.argumentList()) : Collections.emptyList(); + return new FunctionCallExpr(fc.IDENTIFIER().getText(), args, methods); + } + + if (primary.additiveExpression() != null) { + final Expr inner = visit(primary.additiveExpression()); + if (methods.isEmpty()) { + return inner; + } + return new ParenChainExpr(inner, methods); + } + + return new MetricExpr(primary.IDENTIFIER().getText(), methods); + } + + private MethodCall visitMethodCallNode(final MALParser.MethodCallContext ctx) { + final String name = ctx.IDENTIFIER().getText(); + final List args = ctx.argumentList() != null + ? visitArgList(ctx.argumentList()) : Collections.emptyList(); + return new MethodCall(name, args); + } + + private List visitArgList(final MALParser.ArgumentListContext ctx) { + final List args = new ArrayList<>(); + for (final MALParser.ArgumentContext argCtx : ctx.argument()) { + args.add(convertArgument(argCtx)); + } + return args; + } + + private Argument convertArgument(final MALParser.ArgumentContext ctx) { + if (ctx.stringList() != null) { + return convertStringList(ctx.stringList()); + } + if (ctx.numberList() != null) { + return convertNumberList(ctx.numberList()); + } + if (ctx.closureExpression() != null) { + return new ClosureVisitor().visitClosure(ctx.closureExpression()); + } + if (ctx.enumRef() != null) { + return new EnumRefArgument( + ctx.enumRef().IDENTIFIER(0).getText(), + ctx.enumRef().IDENTIFIER(1).getText()); + } + if (ctx.STRING() != null) { + return new StringArgument(stripQuotes(ctx.STRING().getText())); + } + if (ctx.boolLiteral() != null) { + return new BoolArgument(ctx.boolLiteral().TRUE() != null); + } + // additiveExpression — nested expression + return new ExprArgument(visit(ctx.additiveExpression())); + } + + private StringListArgument convertStringList(final MALParser.StringListContext ctx) { + final List values = new ArrayList<>(); + ctx.STRING().forEach(s -> values.add(stripQuotes(s.getText()))); + return new StringListArgument(values); + } + + private NumberListArgument convertNumberList(final MALParser.NumberListContext ctx) { + final List values = new ArrayList<>(); + ctx.NUMBER().forEach(n -> values.add(Double.parseDouble(n.getText()))); + return new NumberListArgument(values); + } + } + + /** + * Visitor for closure expressions within MAL. + */ + private static final class ClosureVisitor { + + ClosureArgument visitClosure(final MALParser.ClosureExpressionContext ctx) { + final List params = new ArrayList<>(); + if (ctx.closureParams() != null) { + ctx.closureParams().IDENTIFIER().forEach(id -> params.add(id.getText())); + } + final List body = convertClosureBody(ctx.closureBody()); + return new ClosureArgument(params, body); + } + + private List convertClosureBody( + final MALParser.ClosureBodyContext ctx) { + final List stmts = new ArrayList<>(); + for (final MALParser.ClosureStatementContext stmtCtx : ctx.closureStatement()) { + stmts.add(convertClosureStatement(stmtCtx)); + } + return stmts; + } + + private ClosureStatement convertClosureStatement( + final MALParser.ClosureStatementContext ctx) { + if (ctx.ifStatement() != null) { + return convertIfStatement(ctx.ifStatement()); + } + if (ctx.returnStatement() != null) { + final ClosureExpr value = ctx.returnStatement().closureExpr() != null + ? convertClosureExpr(ctx.returnStatement().closureExpr()) : null; + return new ClosureReturnStatement(value); + } + if (ctx.assignmentStatement() != null) { + final String target = ctx.assignmentStatement().closureFieldAccess().getText(); + final ClosureExpr value = + convertClosureExpr(ctx.assignmentStatement().closureExpr()); + return new ClosureAssignment(target, value); + } + // expressionStatement + return new ClosureExprStatement( + convertClosureExpr(ctx.expressionStatement().closureExpr())); + } + + private ClosureIfStatement convertIfStatement( + final MALParser.IfStatementContext ctx) { + final ClosureCondition condition = convertCondition(ctx.closureCondition()); + final List thenBranch = new ArrayList<>(); + if (ctx.closureBlock(0) != null) { + for (final MALParser.ClosureStatementContext stmtCtx : + ctx.closureBlock(0).closureStatement()) { + thenBranch.add(convertClosureStatement(stmtCtx)); + } + } + List elseBranch = null; + // Check for else-if or else + if (ctx.ifStatement() != null) { + elseBranch = new ArrayList<>(); + elseBranch.add(convertIfStatement(ctx.ifStatement())); + } else if (ctx.closureBlock().size() > 1) { + elseBranch = new ArrayList<>(); + for (final MALParser.ClosureStatementContext stmtCtx : + ctx.closureBlock(1).closureStatement()) { + elseBranch.add(convertClosureStatement(stmtCtx)); + } + } + return new ClosureIfStatement(condition, thenBranch, elseBranch); + } + + private ClosureCondition convertCondition( + final MALParser.ClosureConditionContext ctx) { + return convertConditionOr(ctx.closureConditionOr()); + } + + private ClosureCondition convertConditionOr( + final MALParser.ClosureConditionOrContext ctx) { + ClosureCondition result = convertConditionAnd(ctx.closureConditionAnd(0)); + for (int i = 1; i < ctx.closureConditionAnd().size(); i++) { + result = new MALExpressionModel.ClosureLogical( + result, LogicalOp.OR, convertConditionAnd(ctx.closureConditionAnd(i))); + } + return result; + } + + private ClosureCondition convertConditionAnd( + final MALParser.ClosureConditionAndContext ctx) { + ClosureCondition result = convertConditionPrimary(ctx.closureConditionPrimary(0)); + for (int i = 1; i < ctx.closureConditionPrimary().size(); i++) { + result = new MALExpressionModel.ClosureLogical( + result, LogicalOp.AND, + convertConditionPrimary(ctx.closureConditionPrimary(i))); + } + return result; + } + + private ClosureCondition convertConditionPrimary( + final MALParser.ClosureConditionPrimaryContext ctx) { + if (ctx instanceof MALParser.ConditionEqContext) { + final MALParser.ConditionEqContext eq = (MALParser.ConditionEqContext) ctx; + return new MALExpressionModel.ClosureComparison( + convertClosureExpr(eq.closureExpr(0)), + CompareOp.EQ, + convertClosureExpr(eq.closureExpr(1))); + } + if (ctx instanceof MALParser.ConditionNeqContext) { + final MALParser.ConditionNeqContext neq = (MALParser.ConditionNeqContext) ctx; + return new MALExpressionModel.ClosureComparison( + convertClosureExpr(neq.closureExpr(0)), + CompareOp.NEQ, + convertClosureExpr(neq.closureExpr(1))); + } + if (ctx instanceof MALParser.ConditionGtContext) { + final MALParser.ConditionGtContext gt = (MALParser.ConditionGtContext) ctx; + return new MALExpressionModel.ClosureComparison( + convertClosureExpr(gt.closureExpr(0)), + CompareOp.GT, + convertClosureExpr(gt.closureExpr(1))); + } + if (ctx instanceof MALParser.ConditionLtContext) { + final MALParser.ConditionLtContext lt = (MALParser.ConditionLtContext) ctx; + return new MALExpressionModel.ClosureComparison( + convertClosureExpr(lt.closureExpr(0)), + CompareOp.LT, + convertClosureExpr(lt.closureExpr(1))); + } + if (ctx instanceof MALParser.ConditionNotContext) { + final MALParser.ConditionNotContext not = (MALParser.ConditionNotContext) ctx; + return new MALExpressionModel.ClosureNot( + convertConditionPrimary(not.closureConditionPrimary())); + } + if (ctx instanceof MALParser.ConditionInContext) { + final MALParser.ConditionInContext in = (MALParser.ConditionInContext) ctx; + final List values = new ArrayList<>(); + if (in.closureListLiteral() != null) { + in.closureListLiteral().STRING().forEach( + s -> values.add(stripQuotes(s.getText()))); + } + return new MALExpressionModel.ClosureInCondition( + convertClosureExpr(in.closureExpr()), values); + } + if (ctx instanceof MALParser.ConditionParenContext) { + final MALParser.ConditionParenContext paren = + (MALParser.ConditionParenContext) ctx; + return convertCondition(paren.closureCondition()); + } + // conditionExpr + final MALParser.ConditionExprContext exprCtx = + (MALParser.ConditionExprContext) ctx; + return new ClosureExprCondition(convertClosureExpr(exprCtx.closureExpr())); + } + + private ClosureExpr convertClosureExpr(final MALParser.ClosureExprContext ctx) { + if (ctx instanceof MALParser.ClosureAddContext) { + final MALParser.ClosureAddContext add = (MALParser.ClosureAddContext) ctx; + return new ClosureBinaryExpr( + convertClosureExpr(add.closureExpr(0)), + ArithmeticOp.ADD, + convertClosureExpr(add.closureExpr(1))); + } + if (ctx instanceof MALParser.ClosureSubContext) { + final MALParser.ClosureSubContext sub = (MALParser.ClosureSubContext) ctx; + return new ClosureBinaryExpr( + convertClosureExpr(sub.closureExpr(0)), + ArithmeticOp.SUB, + convertClosureExpr(sub.closureExpr(1))); + } + if (ctx instanceof MALParser.ClosureMulContext) { + final MALParser.ClosureMulContext mul = (MALParser.ClosureMulContext) ctx; + return new ClosureBinaryExpr( + convertClosureExpr(mul.closureExpr(0)), + ArithmeticOp.MUL, + convertClosureExpr(mul.closureExpr(1))); + } + if (ctx instanceof MALParser.ClosureDivContext) { + final MALParser.ClosureDivContext div = (MALParser.ClosureDivContext) ctx; + return new ClosureBinaryExpr( + convertClosureExpr(div.closureExpr(0)), + ArithmeticOp.DIV, + convertClosureExpr(div.closureExpr(1))); + } + // closurePrimary + final MALParser.ClosurePrimaryContext primary = + (MALParser.ClosurePrimaryContext) ctx; + return convertClosureExprPrimary(primary.closureExprPrimary()); + } + + private ClosureExpr convertClosureExprPrimary( + final MALParser.ClosureExprPrimaryContext ctx) { + if (ctx instanceof MALParser.ClosureStringContext) { + return new ClosureStringLiteral( + stripQuotes(((MALParser.ClosureStringContext) ctx).STRING().getText())); + } + if (ctx instanceof MALParser.ClosureNumberContext) { + return new ClosureNumberLiteral( + Double.parseDouble( + ((MALParser.ClosureNumberContext) ctx).NUMBER().getText())); + } + if (ctx instanceof MALParser.ClosureNullContext) { + return new ClosureNullLiteral(); + } + if (ctx instanceof MALParser.ClosureBoolContext) { + final MALParser.ClosureBoolContext bc = (MALParser.ClosureBoolContext) ctx; + return new ClosureBoolLiteral(bc.boolLiteral().TRUE() != null); + } + // closureChain + final MALParser.ClosureChainContext chain = (MALParser.ClosureChainContext) ctx; + return convertClosureMethodChain(chain.closureMethodChain()); + } + + private ClosureMethodChain convertClosureMethodChain( + final MALParser.ClosureMethodChainContext ctx) { + final String target = ctx.closureTarget().IDENTIFIER().getText(); + final List segments = new ArrayList<>(); + + for (final MALParser.ClosureChainSegmentContext seg : ctx.closureChainSegment()) { + final boolean safeNav = segments.size() < ctx.closureChainSegment().size() + && ctx.safeNav().size() > 0; + // Determine if this segment uses safe navigation + // by checking position relative to safeNav tokens + segments.add(convertClosureChainSegment(seg, false)); + } + + return new ClosureMethodChain(target, segments); + } + + private ClosureChainSegment convertClosureChainSegment( + final MALParser.ClosureChainSegmentContext ctx, + final boolean safeNav) { + if (ctx instanceof MALParser.ChainMethodCallContext) { + final MALParser.ChainMethodCallContext mc = + (MALParser.ChainMethodCallContext) ctx; + final List args = new ArrayList<>(); + if (mc.closureArgList() != null) { + for (final MALParser.ClosureExprContext argCtx : + mc.closureArgList().closureExpr()) { + args.add(convertClosureExpr(argCtx)); + } + } + return new ClosureMethodCallSeg(mc.IDENTIFIER().getText(), args, safeNav); + } + if (ctx instanceof MALParser.ChainIndexAccessContext) { + final MALParser.ChainIndexAccessContext idx = + (MALParser.ChainIndexAccessContext) ctx; + return new ClosureIndexAccess(convertClosureExpr(idx.closureExpr())); + } + // chainFieldAccess + final MALParser.ChainFieldAccessContext fa = + (MALParser.ChainFieldAccessContext) ctx; + return new ClosureFieldAccess(fa.IDENTIFIER().getText(), safeNav); + } + } + + static String stripQuotes(final String s) { + if (s.length() >= 2 && (s.charAt(0) == '\'' || s.charAt(0) == '"')) { + return s.substring(1, s.length() - 1); + } + return s; + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/rt/MalExpressionPackageHolder.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/rt/MalExpressionPackageHolder.java new file mode 100644 index 000000000000..0d468068f78a --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/rt/MalExpressionPackageHolder.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.compiler.rt; + +/** + * Empty marker class used as the class loading anchor for Javassist + * {@code CtClass.toClass(Class)} on JDK 16+. + * Generated MAL expression classes are loaded in this package. + */ +public class MalExpressionPackageHolder { +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java index e723b6add348..36dcb3426ab2 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java @@ -13,37 +13,22 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - * */ package org.apache.skywalking.oap.meter.analyzer.dsl; -import com.google.common.collect.ImmutableList; -import groovy.lang.Binding; -import groovy.lang.GString; -import groovy.lang.GroovyShell; -import groovy.util.DelegatingScript; -import java.lang.reflect.Array; -import java.util.List; -import java.util.Map; - -import org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry; -import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType; -import org.apache.skywalking.oap.server.core.analysis.Layer; -import org.apache.skywalking.oap.server.core.source.DetectPoint; -import org.codehaus.groovy.ast.stmt.DoWhileStatement; -import org.codehaus.groovy.ast.stmt.ForStatement; -import org.codehaus.groovy.ast.stmt.Statement; -import org.codehaus.groovy.ast.stmt.WhileStatement; -import org.codehaus.groovy.control.CompilerConfiguration; -import org.codehaus.groovy.control.customizers.ImportCustomizer; -import org.codehaus.groovy.control.customizers.SecureASTCustomizer; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALClassGenerator; /** - * DSL combines methods to parse groovy based DSL expression. + * DSL compiles MAL expression strings into {@link Expression} objects + * using ANTLR4 parsing and Javassist bytecode generation. */ +@Slf4j public final class DSL { + private static final MALClassGenerator GENERATOR = new MALClassGenerator(); + /** * Parse string literal to Expression object, which can be reused. * @@ -52,40 +37,13 @@ public final class DSL { * @return Expression object could be executed. */ public static Expression parse(final String metricName, final String expression) { - CompilerConfiguration cc = new CompilerConfiguration(); - cc.setScriptBaseClass(DelegatingScript.class.getName()); - ImportCustomizer icz = new ImportCustomizer(); - icz.addImport("K8sRetagType", K8sRetagType.class.getName()); - icz.addImport("DetectPoint", DetectPoint.class.getName()); - icz.addImport("Layer", Layer.class.getName()); - icz.addImport("ProcessRegistry", ProcessRegistry.class.getName()); - cc.addCompilationCustomizers(icz); - - final SecureASTCustomizer secureASTCustomizer = new SecureASTCustomizer(); - secureASTCustomizer.setDisallowedStatements( - ImmutableList.>builder() - .add(WhileStatement.class) - .add(DoWhileStatement.class) - .add(ForStatement.class) - .build()); - // noinspection rawtypes - secureASTCustomizer.setAllowedReceiversClasses( - ImmutableList.builder() - .add(Object.class) - .add(Map.class) - .add(List.class) - .add(Array.class) - .add(K8sRetagType.class) - .add(DetectPoint.class) - .add(Layer.class) - .add(ProcessRegistry.class) - .add(GString.class) - .add(String.class) - .build()); - cc.addCompilationCustomizers(secureASTCustomizer); - - GroovyShell sh = new GroovyShell(new Binding(), cc); - DelegatingScript script = (DelegatingScript) sh.parse(expression); - return new Expression(metricName, expression, script); + try { + final MalExpression malExpr = GENERATOR.compile(metricName, expression); + return new Expression(metricName, expression, malExpr); + } catch (Exception e) { + throw new IllegalStateException( + "Failed to compile MAL expression for metric: " + metricName + + ", expression: " + expression, e); + } } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java index 02912adb05f0..2701f8337c7a 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java @@ -13,60 +13,45 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - * */ package org.apache.skywalking.oap.meter.analyzer.dsl; -import com.google.common.collect.ImmutableMap; -import groovy.lang.ExpandoMetaClass; -import groovy.lang.GroovyObjectSupport; -import groovy.util.DelegatingScript; -import java.time.Instant; import java.util.Map; -import lombok.RequiredArgsConstructor; import lombok.ToString; import lombok.extern.slf4j.Slf4j; /** - * Expression is a reusable monadic container type which represents a DSL expression. + * Same-FQCN replacement for upstream Expression. + * Wraps a compiled {@link MalExpression} (pure Java) instead of a Groovy DelegatingScript. */ @Slf4j @ToString(of = {"literal"}) public class Expression { - private static final ThreadLocal> PROPERTY_REPOSITORY = new ThreadLocal<>(); private final String metricName; - private final String literal; + private final MalExpression expression; - private final DelegatingScript expression; - - public Expression(final String metricName, final String literal, final DelegatingScript expression) { + public Expression(final String metricName, final String literal, final MalExpression expression) { this.metricName = metricName; this.literal = literal; this.expression = expression; - this.empower(); } /** - * Parse the expression statically. - * - * @return Parsed context of the expression. + * Returns compile-time metadata extracted from the expression AST. */ - public ExpressionParsingContext parse() { - try (ExpressionParsingContext ctx = ExpressionParsingContext.create()) { - Result r = run(ImmutableMap.of()); - if (!r.isSuccess() && r.isThrowable()) { - throw new ExpressionParsingException( - "failed to parse expression: " + literal + ", error:" + r.getError()); - } - if (log.isDebugEnabled()) { - log.debug("\"{}\" is parsed", literal); - } - ctx.validate(literal); - return ctx; + public ExpressionMetadata parse() { + final ExpressionMetadata metadata = expression.metadata(); + if (metadata.getScopeType() == null) { + throw new ExpressionParsingException( + literal + ": one of service(), instance() or endpoint() should be invoke"); + } + if (log.isDebugEnabled()) { + log.debug("\"{}\" is parsed", literal); } + return metadata; } /** @@ -76,14 +61,16 @@ public ExpressionParsingContext parse() { * @return The result of execution. */ public Result run(final Map sampleFamilies) { - PROPERTY_REPOSITORY.set(sampleFamilies); try { - SampleFamily sf = (SampleFamily) expression.run(); + for (final SampleFamily s : sampleFamilies.values()) { + if (s != SampleFamily.EMPTY) { + s.context.setMetricName(metricName); + } + } + final SampleFamily sf = expression.run(sampleFamilies); if (sf == SampleFamily.EMPTY) { - if (!ExpressionParsingContext.get().isPresent()) { - if (log.isDebugEnabled()) { - log.debug("result of {} is empty by \"{}\"", sampleFamilies, literal); - } + if (log.isDebugEnabled()) { + log.debug("result of {} is empty by \"{}\"", sampleFamilies, literal); } return Result.fail("Parsed result is an EMPTY sample family"); } @@ -91,62 +78,6 @@ public Result run(final Map sampleFamilies) { } catch (Throwable t) { log.error("failed to run \"{}\"", literal, t); return Result.fail(t); - } finally { - PROPERTY_REPOSITORY.remove(); } } - - private void empower() { - expression.setDelegate(new ExpressionDelegate(metricName, literal)); - extendNumber(Number.class); - } - - private void extendNumber(Class clazz) { - ExpandoMetaClass expando = new ExpandoMetaClass(clazz, true, false); - expando.registerInstanceMethod("plus", new NumberClosure(this, (n, s) -> s.plus(n))); - expando.registerInstanceMethod("minus", new NumberClosure(this, (n, s) -> s.minus(n).negative())); - expando.registerInstanceMethod("multiply", new NumberClosure(this, (n, s) -> s.multiply(n))); - expando.registerInstanceMethod("div", new NumberClosure(this, (n, s) -> s.newValue(v -> n.doubleValue() / v))); - expando.initialize(); - } - - @RequiredArgsConstructor - @SuppressWarnings("unused") // used in MAL expressions - private static class ExpressionDelegate extends GroovyObjectSupport { - public static final DownsamplingType AVG = DownsamplingType.AVG; - public static final DownsamplingType SUM = DownsamplingType.SUM; - public static final DownsamplingType LATEST = DownsamplingType.LATEST; - public static final DownsamplingType SUM_PER_MIN = DownsamplingType.SUM_PER_MIN; - public static final DownsamplingType MAX = DownsamplingType.MAX; - public static final DownsamplingType MIN = DownsamplingType.MIN; - - private final String metricName; - private final String literal; - - public SampleFamily propertyMissing(String sampleName) { - ExpressionParsingContext.get().ifPresent(ctx -> { - if (!ctx.samples.contains(sampleName)) { - ctx.samples.add(sampleName); - } - }); - Map sampleFamilies = PROPERTY_REPOSITORY.get(); - if (sampleFamilies == null) { - return SampleFamily.EMPTY; - } - if (sampleFamilies.containsKey(sampleName)) { - SampleFamily sampleFamily = sampleFamilies.get(sampleName); - sampleFamily.context.setMetricName(this.metricName); - return sampleFamily; - } - if (ExpressionParsingContext.get().isEmpty()) { - log.warn("{} referred by \"{}\" doesn't exist in {}", sampleName, literal, sampleFamilies.keySet()); - } - return SampleFamily.EMPTY; - } - - public Number time() { - return Instant.now().getEpochSecond(); - } - - } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionMetadata.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionMetadata.java new file mode 100644 index 000000000000..11ebbcc7af29 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionMetadata.java @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import lombok.Getter; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +/** + * Immutable metadata extracted from a MAL expression at compile time. + * Replaces the ThreadLocal-based {@code ExpressionParsingContext} pattern. + */ +@Getter +public class ExpressionMetadata { + + private final List samples; + private final ScopeType scopeType; + private final Set scopeLabels; + private final Set aggregationLabels; + private final DownsamplingType downsampling; + private final boolean isHistogram; + private final int[] percentiles; + + public ExpressionMetadata(final List samples, + final ScopeType scopeType, + final Set scopeLabels, + final Set aggregationLabels, + final DownsamplingType downsampling, + final boolean isHistogram, + final int[] percentiles) { + this.samples = Collections.unmodifiableList(samples); + this.scopeType = scopeType; + this.scopeLabels = Collections.unmodifiableSet(scopeLabels); + this.aggregationLabels = Collections.unmodifiableSet(aggregationLabels); + this.downsampling = downsampling; + this.isHistogram = isHistogram; + this.percentiles = percentiles; + } + + /** + * Get labels not related to scope (aggregation labels minus scope labels). + */ + public List getLabels() { + final List result = new ArrayList<>(aggregationLabels); + result.removeAll(scopeLabels); + return result; + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java index 9b532bf72343..b92318d9d372 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java @@ -13,38 +13,66 @@ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. - * */ package org.apache.skywalking.oap.meter.analyzer.dsl; -import groovy.lang.Closure; -import groovy.lang.GroovyShell; +import java.io.IOException; +import java.io.InputStream; import java.util.HashMap; import java.util.Map; import java.util.Objects; +import java.util.Properties; +import java.util.concurrent.atomic.AtomicInteger; import lombok.ToString; import lombok.extern.slf4j.Slf4j; +/** + * Same-FQCN replacement for upstream FilterExpression. + * Loads transpiled {@link MalFilter} classes from mal-filter-expressions.properties + * manifest instead of Groovy filter closures -- no Groovy runtime needed. + */ @Slf4j @ToString(of = {"literal"}) public class FilterExpression { + private static final String MANIFEST_PATH = "META-INF/mal-filter-expressions.properties"; + private static volatile Map FILTER_MAP; + private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + private final String literal; - private final Closure filterClosure; + private final MalFilter malFilter; @SuppressWarnings("unchecked") public FilterExpression(final String literal) { this.literal = literal; - GroovyShell sh = new GroovyShell(); - filterClosure = (Closure) sh.evaluate(literal); + final Map filterMap = loadManifest(); + final String className = filterMap.get(literal); + if (className == null) { + throw new IllegalStateException( + "Transpiled MAL filter not found for: " + literal + + ". Available filters: " + filterMap.size()); + } + + try { + final Class filterClass = Class.forName(className); + malFilter = (MalFilter) filterClass.getDeclaredConstructor().newInstance(); + final int count = LOADED_COUNT.incrementAndGet(); + log.debug("Loaded transpiled MAL filter [{}/{}]: {}", count, filterMap.size(), literal); + } catch (ClassNotFoundException e) { + throw new IllegalStateException( + "Transpiled MAL filter class not found: " + className, e); + } catch (ReflectiveOperationException e) { + throw new IllegalStateException( + "Failed to instantiate transpiled MAL filter: " + className, e); + } } public Map filter(final Map sampleFamilies) { try { - Map result = new HashMap<>(); - for (Map.Entry entry : sampleFamilies.entrySet()) { - SampleFamily afterFilter = entry.getValue().filter(filterClosure); + final Map result = new HashMap<>(); + for (final Map.Entry entry : sampleFamilies.entrySet()) { + final SampleFamily afterFilter = entry.getValue().filter(malFilter::test); if (!Objects.equals(afterFilter, SampleFamily.EMPTY)) { result.put(entry.getKey(), afterFilter); } @@ -55,4 +83,31 @@ public Map filter(final Map sampleFa } return sampleFamilies; } + + private static Map loadManifest() { + if (FILTER_MAP != null) { + return FILTER_MAP; + } + synchronized (FilterExpression.class) { + if (FILTER_MAP != null) { + return FILTER_MAP; + } + final Map map = new HashMap<>(); + try (InputStream is = FilterExpression.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { + if (is == null) { + log.warn("MAL filter manifest not found: {}", MANIFEST_PATH); + FILTER_MAP = map; + return map; + } + final Properties props = new Properties(); + props.load(is); + props.forEach((k, v) -> map.put((String) k, (String) v)); + } catch (IOException e) { + throw new IllegalStateException("Failed to load MAL filter manifest", e); + } + log.info("Loaded {} transpiled MAL filters from manifest", map.size()); + FILTER_MAP = map; + return map; + } + } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java index cfaf1d5beee5..2400cf319a4f 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java @@ -22,9 +22,14 @@ /** * Pure Java replacement for Groovy-based MAL DelegatingScript. - * Each transpiled MAL expression implements this interface. + * Each compiled MAL expression implements this interface. */ -@FunctionalInterface public interface MalExpression { SampleFamily run(Map samples); + + /** + * Returns compile-time metadata extracted from the expression AST: + * sample names, scope type, aggregation labels, downsampling, etc. + */ + ExpressionMetadata metadata(); } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java index 5a979d0a381d..12fc44847e63 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java @@ -57,7 +57,6 @@ import com.google.common.base.Strings; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Maps; -import groovy.lang.Closure; import io.vavr.Function2; import io.vavr.Function3; import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.DecorateFunction; @@ -223,7 +222,6 @@ public SampleFamily min(List by) { } public SampleFamily avg(List by) { - ExpressionParsingContext.get().ifPresent(ctx -> ctx.aggregationLabels.addAll(by)); if (this == EMPTY) { return EMPTY; } @@ -249,7 +247,6 @@ public SampleFamily avg(List by) { } public SampleFamily count(List by) { - ExpressionParsingContext.get().ifPresent(ctx -> ctx.aggregationLabels.addAll(by)); if (this == EMPTY) { return EMPTY; } @@ -302,7 +299,6 @@ public SampleFamily count(List by) { } protected SampleFamily aggregate(List by, DoubleBinaryOperator aggregator) { - ExpressionParsingContext.get().ifPresent(ctx -> ctx.aggregationLabels.addAll(by)); if (this == EMPTY) { return EMPTY; } @@ -382,29 +378,6 @@ public SampleFamily irate() { ); } - @SuppressWarnings(value = "unchecked") - public SampleFamily tag(Closure cl) { - if (this == EMPTY) { - return EMPTY; - } - return SampleFamily.build( - this.context, - Arrays.stream(samples) - .map(sample -> { - Object delegate = new Object(); - Closure c = cl.rehydrate(delegate, sample, delegate); - Map arg = Maps.newHashMap(sample.labels); - Object r = c.call(arg); - return sample.toBuilder() - .labels( - ImmutableMap.copyOf( - Optional.ofNullable((r instanceof Map) ? (Map) r : null) - .orElse(arg))) - .build(); - }).toArray(Sample[]::new) - ); - } - @SuppressWarnings(value = "unchecked") public SampleFamily tag(TagFunction fn) { if (this == EMPTY) { @@ -425,19 +398,6 @@ public SampleFamily tag(TagFunction fn) { ); } - public SampleFamily filter(Closure filter) { - if (this == EMPTY) { - return EMPTY; - } - final Sample[] filtered = Arrays.stream(samples) - .filter(it -> filter.call(it.labels)) - .toArray(Sample[]::new); - if (filtered.length == 0) { - return EMPTY; - } - return SampleFamily.build(context, filtered); - } - public SampleFamily filter(SampleFilter filter) { if (this == EMPTY) { return EMPTY; @@ -478,7 +438,6 @@ public SampleFamily histogram(String le) { public SampleFamily histogram(String le, TimeUnit unit) { long scale = unit.toMillis(1); Preconditions.checkArgument(scale > 0); - ExpressionParsingContext.get().ifPresent(ctx -> ctx.isHistogram = true); if (this == EMPTY) { return EMPTY; } @@ -508,21 +467,11 @@ public SampleFamily histogram(String le, TimeUnit unit) { public SampleFamily histogram_percentile(List percentiles) { Preconditions.checkArgument(percentiles.size() > 0); - int[] p = percentiles.stream().mapToInt(i -> i).toArray(); - ExpressionParsingContext.get().ifPresent(ctx -> { - Preconditions.checkState( - ctx.isHistogram, "histogram() should be invoked before invoking histogram_percentile()"); - ctx.percentiles = p; - }); return this; } public SampleFamily service(List labelKeys, Layer layer) { Preconditions.checkArgument(labelKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.SERVICE; - ctx.scopeLabels.addAll(labelKeys); - }); if (this == EMPTY) { return EMPTY; } @@ -531,44 +480,17 @@ public SampleFamily service(List labelKeys, Layer layer) { public SampleFamily service(List labelKeys, String delimiter, Layer layer) { Preconditions.checkArgument(labelKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.SERVICE; - ctx.scopeLabels.addAll(labelKeys); - }); if (this == EMPTY) { return EMPTY; } return createMeterSamples(new ServiceEntityDescription(labelKeys, layer, delimiter)); } - public SampleFamily instance(List serviceKeys, String serviceDelimiter, - List instanceKeys, String instanceDelimiter, - Layer layer, Closure> propertiesExtractor) { - Preconditions.checkArgument(serviceKeys.size() > 0); - Preconditions.checkArgument(instanceKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.SERVICE_INSTANCE; - ctx.scopeLabels.addAll(serviceKeys); - ctx.scopeLabels.addAll(instanceKeys); - }); - if (this == EMPTY) { - return EMPTY; - } - return createMeterSamples(new InstanceEntityDescription( - serviceKeys, instanceKeys, layer, serviceDelimiter, instanceDelimiter, - propertiesExtractor == null ? null : propertiesExtractor::call)); - } - public SampleFamily instance(List serviceKeys, String serviceDelimiter, List instanceKeys, String instanceDelimiter, Layer layer, PropertiesExtractor propertiesExtractor) { Preconditions.checkArgument(serviceKeys.size() > 0); Preconditions.checkArgument(instanceKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.SERVICE_INSTANCE; - ctx.scopeLabels.addAll(serviceKeys); - ctx.scopeLabels.addAll(instanceKeys); - }); if (this == EMPTY) { return EMPTY; } @@ -577,17 +499,12 @@ public SampleFamily instance(List serviceKeys, String serviceDelimiter, } public SampleFamily instance(List serviceKeys, List instanceKeys, Layer layer) { - return instance(serviceKeys, Const.POINT, instanceKeys, Const.POINT, layer, (Closure>) null); + return instance(serviceKeys, Const.POINT, instanceKeys, Const.POINT, layer, (PropertiesExtractor) null); } public SampleFamily endpoint(List serviceKeys, List endpointKeys, String delimiter, Layer layer) { Preconditions.checkArgument(serviceKeys.size() > 0); Preconditions.checkArgument(endpointKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.ENDPOINT; - ctx.scopeLabels.addAll(serviceKeys); - ctx.scopeLabels.addAll(endpointKeys); - }); if (this == EMPTY) { return EMPTY; } @@ -602,13 +519,6 @@ public SampleFamily process(List serviceKeys, List serviceInstan Preconditions.checkArgument(serviceKeys.size() > 0); Preconditions.checkArgument(serviceInstanceKeys.size() > 0); Preconditions.checkArgument(processKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.PROCESS; - ctx.scopeLabels.addAll(serviceKeys); - ctx.scopeLabels.addAll(serviceInstanceKeys); - ctx.scopeLabels.addAll(processKeys); - ctx.scopeLabels.add(layerKey); - }); if (this == EMPTY) { return EMPTY; } @@ -618,11 +528,6 @@ public SampleFamily process(List serviceKeys, List serviceInstan public SampleFamily serviceRelation(DetectPoint detectPoint, List sourceServiceKeys, List destServiceKeys, Layer layer) { Preconditions.checkArgument(sourceServiceKeys.size() > 0); Preconditions.checkArgument(destServiceKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.SERVICE_RELATION; - ctx.scopeLabels.addAll(sourceServiceKeys); - ctx.scopeLabels.addAll(destServiceKeys); - }); if (this == EMPTY) { return EMPTY; } @@ -632,31 +537,12 @@ public SampleFamily serviceRelation(DetectPoint detectPoint, List source public SampleFamily serviceRelation(DetectPoint detectPoint, List sourceServiceKeys, List destServiceKeys, String delimiter, Layer layer, String componentIdKey) { Preconditions.checkArgument(sourceServiceKeys.size() > 0); Preconditions.checkArgument(destServiceKeys.size() > 0); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.SERVICE_RELATION; - ctx.scopeLabels.addAll(sourceServiceKeys); - ctx.scopeLabels.addAll(destServiceKeys); - ctx.scopeLabels.add(componentIdKey); - }); if (this == EMPTY) { return EMPTY; } return createMeterSamples(new ServiceRelationEntityDescription(sourceServiceKeys, destServiceKeys, detectPoint, layer, delimiter, componentIdKey)); } - public SampleFamily forEach(List array, Closure each) { - if (this == EMPTY) { - return EMPTY; - } - return SampleFamily.build(this.context, Arrays.stream(this.samples).map(sample -> { - Map labels = Maps.newHashMap(sample.getLabels()); - for (String element : array) { - each.call(element, labels); - } - return sample.toBuilder().labels(ImmutableMap.copyOf(labels)).build(); - }).toArray(Sample[]::new)); - } - public SampleFamily forEach(List array, ForEachFunction each) { if (this == EMPTY) { return EMPTY; @@ -675,15 +561,6 @@ public SampleFamily processRelation(String detectPointKey, List serviceK Preconditions.checkArgument(instanceKeys.size() > 0); Preconditions.checkArgument(StringUtil.isNotEmpty(sourceProcessIdKey)); Preconditions.checkArgument(StringUtil.isNotEmpty(destProcessIdKey)); - ExpressionParsingContext.get().ifPresent(ctx -> { - ctx.scopeType = ScopeType.PROCESS_RELATION; - ctx.scopeLabels.addAll(serviceKeys); - ctx.scopeLabels.addAll(instanceKeys); - ctx.scopeLabels.add(detectPointKey); - ctx.scopeLabels.add(sourceProcessIdKey); - ctx.scopeLabels.add(destProcessIdKey); - ctx.scopeLabels.add(componentKey); - }); if (this == EMPTY) { return EMPTY; } @@ -758,43 +635,10 @@ private SampleFamily newValue(SampleFamily another, Function2 it.downsampling = type); - return this; - } - - /** - * Decorate the service meter entity with the given closure. - */ - public SampleFamily decorate(Closure c) { - ExpressionParsingContext.get().ifPresent(ctx -> { - if (ctx.getScopeType() != ScopeType.SERVICE) { - throw new IllegalStateException("decorate() should be invoked after service()"); - } - if (ctx.isHistogram()) { - throw new IllegalStateException("decorate() not supported for histogram metrics"); - } - }); - if (this == EMPTY) { - return EMPTY; - } - this.context.getMeterSamples().keySet().forEach(meterEntity -> { - // Only service meter entity can be decorated - if (meterEntity.getScopeType().equals(ScopeType.SERVICE)) { - c.call(meterEntity); - } - }); return this; } public SampleFamily decorate(DecorateFunction c) { - ExpressionParsingContext.get().ifPresent(ctx -> { - if (ctx.getScopeType() != ScopeType.SERVICE) { - throw new IllegalStateException("decorate() should be invoked after service()"); - } - if (ctx.isHistogram()) { - throw new IllegalStateException("decorate() not supported for histogram metrics"); - } - }); if (this == EMPTY) { return EMPTY; } diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java new file mode 100644 index 000000000000..3e6cbbb8f50c --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.compiler; + +import javassist.ClassPool; +import org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertNotNull; + +class MALClassGeneratorTest { + + private MALClassGenerator generator; + + @BeforeEach + void setUp() { + generator = new MALClassGenerator(new ClassPool(true)); + } + + @Test + void compileSimpleMetric() throws Exception { + final MalExpression expr = generator.compile( + "test_metric", "instance_jvm_cpu"); + assertNotNull(expr); + // Run returns SampleFamily.EMPTY since no samples are provided + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileMethodChain() throws Exception { + final MalExpression expr = generator.compile( + "test_sum", + "instance_jvm_cpu.sum(['service', 'instance'])"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileArithmeticAdd() throws Exception { + final MalExpression expr = generator.compile( + "test_add", "metric_a + metric_b"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileNumberTimesMetric() throws Exception { + final MalExpression expr = generator.compile( + "test_mul", "100 * process_cpu_seconds_total"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileParenChainExpr() throws Exception { + final MalExpression expr = generator.compile( + "test_paren", + "(process_cpu_seconds_total * 100).sum(['service', 'instance']).rate('PT1M')"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileWithEnumRef() throws Exception { + final MalExpression expr = generator.compile( + "test_enum", + "instance_jvm_cpu.sum(['service']).service(['service'], Layer.GENERAL)"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileWithDownsamplingType() throws Exception { + final MalExpression expr = generator.compile( + "test_ds", + "instance_jvm_cpu.sum(['service']).downsampling(SUM)"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void compileWithClosureTag() throws Exception { + final MalExpression expr = generator.compile( + "test_closure", + "instance_jvm_cpu.tag({tags -> tags.service_name = 'svc1'})"); + assertNotNull(expr); + assertNotNull(expr.run(java.util.Map.of())); + } + + @Test + void generateSourceReturnsJavaCode() { + final String source = generator.generateSource( + "instance_jvm_cpu.sum(['service'])"); + assertNotNull(source); + // Generated source should contain getOrDefault for the metric + org.junit.jupiter.api.Assertions.assertTrue( + source.contains("getOrDefault")); + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParserTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParserTest.java new file mode 100644 index 000000000000..40f50f574508 --- /dev/null +++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParserTest.java @@ -0,0 +1,238 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.meter.analyzer.compiler; + +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.BinaryExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ClosureArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.EnumRefArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ExprArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.MetricExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.NumberExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.NumberListArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.ParenChainExpr; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.StringArgument; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALExpressionModel.StringListArgument; +import org.junit.jupiter.api.Test; + +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertThrows; + +class MALScriptParserTest { + + @Test + void parseSimpleMetric() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "instance_golang_heap_alloc"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals("instance_golang_heap_alloc", metric.getMetricName()); + assertEquals(0, metric.getMethodChain().size()); + } + + @Test + void parseMethodChain() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "jvm_memory_bytes_used.sum(['service', 'host_name', 'area'])"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals("jvm_memory_bytes_used", metric.getMetricName()); + assertEquals(1, metric.getMethodChain().size()); + assertEquals("sum", metric.getMethodChain().get(0).getName()); + + final StringListArgument sl = + (StringListArgument) metric.getMethodChain().get(0).getArguments().get(0); + assertEquals(List.of("service", "host_name", "area"), sl.getValues()); + } + + @Test + void parseTagEqualWithRateAndService() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "mysql_global_status_commands_total" + + ".tagEqual('command','insert')" + + ".sum(['service_instance_id','host_name'])" + + ".rate('PT1M')"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(3, metric.getMethodChain().size()); + assertEquals("tagEqual", metric.getMethodChain().get(0).getName()); + assertEquals("sum", metric.getMethodChain().get(1).getName()); + assertEquals("rate", metric.getMethodChain().get(2).getName()); + + // Check tagEqual arguments + final StringArgument arg0 = + (StringArgument) metric.getMethodChain().get(0).getArguments().get(0); + assertEquals("command", arg0.getValue()); + final StringArgument arg1 = + (StringArgument) metric.getMethodChain().get(0).getArguments().get(1); + assertEquals("insert", arg1.getValue()); + } + + @Test + void parseHistogramPercentile() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "metric.sum(['le']).histogram().histogram_percentile([50,75,90,95,99])"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(3, metric.getMethodChain().size()); + assertEquals("histogram", metric.getMethodChain().get(1).getName()); + assertEquals("histogram_percentile", metric.getMethodChain().get(2).getName()); + + final NumberListArgument nl = + (NumberListArgument) metric.getMethodChain().get(2).getArguments().get(0); + assertEquals(List.of(50.0, 75.0, 90.0, 95.0, 99.0), nl.getValues()); + } + + @Test + void parseArithmeticAdd() { + final MALExpressionModel.Expr ast = MALScriptParser.parse("metric1 + metric2"); + assertInstanceOf(BinaryExpr.class, ast); + final BinaryExpr bin = (BinaryExpr) ast; + assertEquals(MALExpressionModel.ArithmeticOp.ADD, bin.getOp()); + assertEquals("metric1", ((MetricExpr) bin.getLeft()).getMetricName()); + assertEquals("metric2", ((MetricExpr) bin.getRight()).getMetricName()); + } + + @Test + void parseArithmeticMultiply() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "(process_cpu_seconds_total * 100).sum(['service', 'host_name']).rate('PT1M')"); + assertInstanceOf(ParenChainExpr.class, ast); + final ParenChainExpr parenChain = (ParenChainExpr) ast; + + // Inner expression is (metric * 100) + assertInstanceOf(BinaryExpr.class, parenChain.getInner()); + final BinaryExpr inner = (BinaryExpr) parenChain.getInner(); + assertEquals(MALExpressionModel.ArithmeticOp.MUL, inner.getOp()); + assertEquals("process_cpu_seconds_total", ((MetricExpr) inner.getLeft()).getMetricName()); + assertEquals(100.0, ((NumberExpr) inner.getRight()).getValue()); + + // Method chain: .sum(['service', 'host_name']).rate('PT1M') + assertEquals(2, parenChain.getMethodChain().size()); + assertEquals("sum", parenChain.getMethodChain().get(0).getName()); + assertEquals("rate", parenChain.getMethodChain().get(1).getName()); + } + + @Test + void parseNumberTimesMetric() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "100 * metrics_aggregation_queue_used_percentage" + + ".sum(['service', 'host_name', 'level', 'slot'])"); + assertInstanceOf(BinaryExpr.class, ast); + final BinaryExpr bin = (BinaryExpr) ast; + assertEquals(MALExpressionModel.ArithmeticOp.MUL, bin.getOp()); + assertInstanceOf(NumberExpr.class, bin.getLeft()); + assertEquals(100.0, ((NumberExpr) bin.getLeft()).getValue()); + assertInstanceOf(MetricExpr.class, bin.getRight()); + } + + @Test + void parseEnumRefArgument() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "metric.service(['svc'], Layer.GENERAL)"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + final EnumRefArgument enumRef = + (EnumRefArgument) metric.getMethodChain().get(0).getArguments().get(1); + assertEquals("Layer", enumRef.getEnumType()); + assertEquals("GENERAL", enumRef.getEnumValue()); + } + + @Test + void parseDownsampling() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "metric.histogram().histogram_percentile([50,75,90,95,99]).downsampling(SUM)"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(3, metric.getMethodChain().size()); + assertEquals("downsampling", metric.getMethodChain().get(2).getName()); + // SUM is parsed as an expression argument (identifier) + // In the grammar, it matches additiveExpression -> metric reference + final ExprArgument exprArg = + (ExprArgument) metric.getMethodChain().get(2).getArguments().get(0); + assertInstanceOf(MetricExpr.class, exprArg.getExpr()); + assertEquals("SUM", ((MetricExpr) exprArg.getExpr()).getMetricName()); + } + + @Test + void parseValueEqual() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "kube_node_status_condition.valueEqual(1).sum(['cluster','node','condition'])"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(2, metric.getMethodChain().size()); + assertEquals("valueEqual", metric.getMethodChain().get(0).getName()); + } + + @Test + void parseDivTwoMetrics() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "metric_sum.div(metric_count)"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(1, metric.getMethodChain().size()); + assertEquals("div", metric.getMethodChain().get(0).getName()); + final ExprArgument divArg = + (ExprArgument) metric.getMethodChain().get(0).getArguments().get(0); + assertInstanceOf(MetricExpr.class, divArg.getExpr()); + assertEquals("metric_count", ((MetricExpr) divArg.getExpr()).getMetricName()); + } + + @Test + void parseClosureTag() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "metric.tag({tags -> tags.service_name = 'svc1'})"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(1, metric.getMethodChain().size()); + assertEquals("tag", metric.getMethodChain().get(0).getName()); + + final ClosureArgument closure = + (ClosureArgument) metric.getMethodChain().get(0).getArguments().get(0); + assertEquals(List.of("tags"), closure.getParams()); + assertEquals(1, closure.getBody().size()); + } + + @Test + void parseRetagByK8sMeta() { + final MALExpressionModel.Expr ast = MALScriptParser.parse( + "kube_pod_status_phase" + + ".retagByK8sMeta('service', K8sRetagType.Pod2Service, 'pod', 'namespace')" + + ".tagNotEqual('service', '')" + + ".valueEqual(1)" + + ".sum(['cluster', 'service', 'phase'])"); + assertInstanceOf(MetricExpr.class, ast); + final MetricExpr metric = (MetricExpr) ast; + assertEquals(4, metric.getMethodChain().size()); + assertEquals("retagByK8sMeta", metric.getMethodChain().get(0).getName()); + + // Check K8sRetagType.Pod2Service argument + final EnumRefArgument enumArg = + (EnumRefArgument) metric.getMethodChain().get(0).getArguments().get(1); + assertEquals("K8sRetagType", enumArg.getEnumType()); + assertEquals("Pod2Service", enumArg.getEnumValue()); + } + + @Test + void parseSyntaxErrorThrows() { + assertThrows(IllegalArgumentException.class, + () -> MALScriptParser.parse("metric.sum(")); + } +} diff --git a/oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java similarity index 67% rename from oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java rename to oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java index d64847e77186..0e6dc713b83b 100644 --- a/oap-server/analyzer/meter-analyzer-v2/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java +++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSLV2Test.java @@ -18,7 +18,6 @@ package org.apache.skywalking.oap.meter.analyzer.dsl; import com.google.common.collect.ImmutableMap; -import java.util.HashMap; import java.util.Map; import org.junit.jupiter.api.Test; @@ -31,42 +30,37 @@ class DSLV2Test { @Test - void parseRejectsNullMetricName() { - assertThrows(UnsupportedOperationException.class, () -> DSL.parse(null, "test_metric")); + void parseCompilesSimpleExpression() { + final Expression expr = DSL.parse("test_metric", "test_metric.sum(['service'])"); + assertNotNull(expr); } @Test - void parseThrowsWhenManifestMissing() { - assertThrows(IllegalStateException.class, () -> DSL.parse("nonexistent_metric", "some_expr")); + void parseThrowsOnInvalidExpression() { + assertThrows(IllegalStateException.class, + () -> DSL.parse("bad", "??? invalid !!!")); } @Test - void expressionRunWithMalExpression() { - final MalExpression simple = samples -> - samples.getOrDefault("test_metric", SampleFamily.EMPTY); - - final Expression expr = new Expression("test_metric", "test_metric", simple); + void expressionRunWithCompiledExpression() { + final Expression expr = DSL.parse("test_metric", + "test_metric.service(['service'], Layer.GENERAL)"); // Run with empty map should return fail (EMPTY) final Result emptyResult = expr.run(Map.of()); assertNotNull(emptyResult); assertFalse(emptyResult.isSuccess()); + } - // Run with a real sample should return success - final Sample sample = Sample.builder() - .name("test_metric") - .labels(ImmutableMap.of("service", "svc1")) - .value(42.0) - .timestamp(System.currentTimeMillis()) - .build(); - final SampleFamily sf = SampleFamily.build(SampleFamily.RunningContext.instance(), sample); - final Map sampleMap = new HashMap<>(); - sampleMap.put("test_metric", sf); + @Test + void metadataExtraction() { + final Expression expr = DSL.parse("test_metric", + "test_metric.sum(['service', 'instance']).service(['service'], Layer.GENERAL)"); - final Result result = expr.run(sampleMap); - assertNotNull(result); - assertTrue(result.isSuccess()); - assertEquals(sf, result.getData()); + final ExpressionMetadata metadata = expr.parse(); + assertNotNull(metadata); + assertTrue(metadata.getSamples().contains("test_metric")); + assertNotNull(metadata.getScopeType()); } @Test diff --git a/oap-server/analyzer/pom.xml b/oap-server/analyzer/pom.xml index eacd244e8709..8cad4dff5a9d 100644 --- a/oap-server/analyzer/pom.xml +++ b/oap-server/analyzer/pom.xml @@ -30,17 +30,10 @@ agent-analyzer - log-analyzer - meter-analyzer event-analyzer - mal-transpiler - lal-transpiler - meter-analyzer-v2 - log-analyzer-v2 - hierarchy-v1 - hierarchy-v2 - hierarchy-v1-v2-checker - mal-lal-v1-v2-checker + meter-analyzer + log-analyzer + hierarchy diff --git a/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/resolver/PprofQuery.java b/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/resolver/PprofQuery.java index 9b1096cfbdec..3cd830875c08 100644 --- a/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/resolver/PprofQuery.java +++ b/oap-server/server-query-plugin/query-graphql-plugin/src/main/java/org/apache/skywalking/oap/query/graphql/resolver/PprofQuery.java @@ -19,7 +19,7 @@ package org.apache.skywalking.oap.query.graphql.resolver; import org.apache.skywalking.oap.server.core.CoreModule; -import groovy.util.logging.Slf4j; +import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.server.library.module.ModuleManager; import graphql.kickstart.tools.GraphQLQueryResolver; import org.apache.skywalking.oap.server.core.profiling.pprof.PprofQueryService; diff --git a/pom.xml b/pom.xml index 6e5575ea89ec..9beacb8b7797 100755 --- a/pom.xml +++ b/pom.xml @@ -83,6 +83,7 @@ oap-server oap-server-bom + test/script-compiler diff --git a/oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml b/test/script-compiler/hierarchy-v1-v2-checker/pom.xml similarity index 84% rename from oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml rename to test/script-compiler/hierarchy-v1-v2-checker/pom.xml index a663a0fd1a51..1c753e0f2f45 100644 --- a/oap-server/analyzer/hierarchy-v1-v2-checker/pom.xml +++ b/test/script-compiler/hierarchy-v1-v2-checker/pom.xml @@ -19,25 +19,27 @@ - analyzer + script-compiler org.apache.skywalking ${revision} 4.0.0 hierarchy-v1-v2-checker - Dual-path comparison tests: Groovy hierarchy rules (v1) vs Java hierarchy rules (v2) + Dual-path comparison tests: Groovy hierarchy rules (v1) vs compiler-generated Javassist hierarchy rules (v2) + org.apache.skywalking - hierarchy-v1 + hierarchy-v1-with-groovy ${project.version} test + org.apache.skywalking - hierarchy-v2 + hierarchy ${project.version} test diff --git a/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java b/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java similarity index 100% rename from oap-server/analyzer/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java rename to test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java diff --git a/oap-server/analyzer/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml b/test/script-compiler/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml similarity index 100% rename from oap-server/analyzer/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml rename to test/script-compiler/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml diff --git a/oap-server/analyzer/hierarchy-v1/pom.xml b/test/script-compiler/hierarchy-v1-with-groovy/pom.xml similarity index 94% rename from oap-server/analyzer/hierarchy-v1/pom.xml rename to test/script-compiler/hierarchy-v1-with-groovy/pom.xml index 0e2cf572f295..d4f039cb3aff 100644 --- a/oap-server/analyzer/hierarchy-v1/pom.xml +++ b/test/script-compiler/hierarchy-v1-with-groovy/pom.xml @@ -19,13 +19,13 @@ - analyzer + script-compiler org.apache.skywalking ${revision} 4.0.0 - hierarchy-v1 + hierarchy-v1-with-groovy Groovy-based hierarchy rule provider (for checker module only, not runtime) diff --git a/oap-server/analyzer/hierarchy-v1/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java b/test/script-compiler/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java similarity index 100% rename from oap-server/analyzer/hierarchy-v1/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java rename to test/script-compiler/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java diff --git a/oap-server/analyzer/lal-transpiler/pom.xml b/test/script-compiler/lal-v1-with-groovy/pom.xml similarity index 82% rename from oap-server/analyzer/lal-transpiler/pom.xml rename to test/script-compiler/lal-v1-with-groovy/pom.xml index b722b255fcfc..adae2c5b03bc 100644 --- a/oap-server/analyzer/lal-transpiler/pom.xml +++ b/test/script-compiler/lal-v1-with-groovy/pom.xml @@ -7,25 +7,25 @@ ~ (the "License"); you may not use this file except in compliance with ~ the License. You may obtain a copy of the License at ~ - ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License. - ~ --> - analyzer + script-compiler org.apache.skywalking ${revision} 4.0.0 - lal-transpiler + lal-v1-with-groovy + jar @@ -33,11 +33,6 @@ log-analyzer ${project.version} - - org.apache.skywalking - mal-transpiler - ${project.version} - org.apache.groovy groovy diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java new file mode 100644 index 000000000000..763dd64d28ef --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java @@ -0,0 +1,224 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl; + +import com.google.protobuf.Message; +import groovy.lang.Closure; +import groovy.lang.GroovyObjectSupport; +import groovy.lang.MissingPropertyException; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicReference; +import java.util.regex.Matcher; +import lombok.Getter; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily; +import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.DatabaseSlowStatementBuilder; + +import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.SampledTraceBuilder; +import org.apache.skywalking.oap.server.core.source.Log; + +/** + * The binding bridge between OAP and the DSL, which provides some convenient methods to ease the use of the raw {@link groovy.lang.Binding#setProperty(java.lang.String, java.lang.Object)} and {@link + * groovy.lang.Binding#getProperty(java.lang.String)}. + */ +public class Binding extends groovy.lang.Binding { + public static final String KEY_LOG = "log"; + + public static final String KEY_PARSED = "parsed"; + + public static final String KEY_SAVE = "save"; + + public static final String KEY_ABORT = "abort"; + + public static final String KEY_METRICS_CONTAINER = "metrics_container"; + + public static final String KEY_LOG_CONTAINER = "log_container"; + + public static final String KEY_DATABASE_SLOW_STATEMENT = "database_slow_statement"; + + public static final String KEY_SAMPLED_TRACE = "sampled_trace"; + + public Binding() { + setProperty(KEY_PARSED, new Parsed()); + } + + public Binding log(final LogData.Builder log) { + setProperty(KEY_LOG, log); + setProperty(KEY_SAVE, true); + setProperty(KEY_ABORT, false); + setProperty(KEY_METRICS_CONTAINER, null); + setProperty(KEY_LOG_CONTAINER, null); + parsed().log = log; + return this; + } + + public Binding log(final LogData log) { + return log(log.toBuilder()); + } + + public LogData.Builder log() { + return (LogData.Builder) getProperty(KEY_LOG); + } + + public Binding extraLog(final Message extraLog) { + parsed().extraLog = extraLog; + return this; + } + + public Message extraLog() { + return parsed().getExtraLog(); + } + + public Binding parsed(final Matcher parsed) { + parsed().matcher = parsed; + return this; + } + + public Binding parsed(final Map parsed) { + parsed().map = parsed; + return this; + } + + public Parsed parsed() { + return (Parsed) getProperty(KEY_PARSED); + } + + public DatabaseSlowStatementBuilder databaseSlowStatement() { + return (DatabaseSlowStatementBuilder) getProperty(KEY_DATABASE_SLOW_STATEMENT); + } + + public Binding databaseSlowStatement(DatabaseSlowStatementBuilder databaseSlowStatementBuilder) { + setProperty(KEY_DATABASE_SLOW_STATEMENT, databaseSlowStatementBuilder); + return this; + } + + public SampledTraceBuilder sampledTraceBuilder() { + return (SampledTraceBuilder) getProperty(KEY_SAMPLED_TRACE); + } + + public Binding sampledTrace(SampledTraceBuilder sampledTraceBuilder) { + setProperty(KEY_SAMPLED_TRACE, sampledTraceBuilder); + return this; + } + + public Binding save() { + setProperty(KEY_SAVE, true); + return this; + } + + public Binding drop() { + setProperty(KEY_SAVE, false); + return this; + } + + public boolean shouldSave() { + return (boolean) getProperty(KEY_SAVE); + } + + public Binding abort() { + setProperty(KEY_ABORT, true); + return this; + } + + public boolean shouldAbort() { + return (boolean) getProperty(KEY_ABORT); + } + + /** + * Set the metrics container to store all metrics generated from the pipeline, + * if no container is set, all generated metrics will be sent to MAL engine for further processing, + * if metrics container is set, all metrics are only stored in the container, and won't be sent to MAL. + * + * @param container the metrics container + */ + public Binding metricsContainer(List container) { + setProperty(KEY_METRICS_CONTAINER, container); + return this; + } + + public Optional> metricsContainer() { + // noinspection unchecked + return Optional.ofNullable((List) getProperty(KEY_METRICS_CONTAINER)); + } + + /** + * Set the log container to store the final log if it should be persisted in storage, + * if no container is set, the final log will be sent to source receiver, + * if log container is set, the log is only stored in the container, and won't be sent to source receiver. + * + * @param container the log container + */ + public Binding logContainer(AtomicReference container) { + setProperty(KEY_LOG_CONTAINER, container); + return this; + } + + public Optional> logContainer() { + // noinspection unchecked + return Optional.ofNullable((AtomicReference) getProperty(KEY_LOG_CONTAINER)); + } + + public static class Parsed extends GroovyObjectSupport { + @Getter + private Matcher matcher; + + @Getter + private Map map; + + @Getter + private Message.Builder log; + + @Getter + private Message extraLog; + + public Object getAt(final String key) { + Object result; + if (matcher != null && (result = matcher.group(key)) != null) { + return result; + } + if (map != null && (result = map.get(key)) != null) { + return result; + } + if (extraLog != null && (result = getField(extraLog, key)) != null) { + return result; + } + if (log != null && (result = getField(log, key)) != null) { + return result; + } + return null; + } + + @SuppressWarnings("unused") + public Object propertyMissing(final String name) { + return getAt(name); + } + + static Object getField(Object obj, String name) { + try { + Closure c = new Closure(obj, obj) { + }; + return c.getProperty(name); + } catch (MissingPropertyException ignored) { + } + return null; + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java new file mode 100644 index 000000000000..c5666d8061d0 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl; + +import com.google.common.collect.ImmutableList; + +import groovy.lang.GString; +import groovy.lang.GroovyShell; +import groovy.transform.CompileStatic; +import groovy.util.DelegatingScript; +import java.lang.reflect.Array; +import java.util.List; +import java.util.Map; +import lombok.AccessLevel; +import lombok.RequiredArgsConstructor; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.LALDelegatingScript; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.codehaus.groovy.ast.stmt.DoWhileStatement; +import org.codehaus.groovy.ast.stmt.ForStatement; +import org.codehaus.groovy.ast.stmt.Statement; +import org.codehaus.groovy.ast.stmt.WhileStatement; +import org.codehaus.groovy.control.CompilerConfiguration; +import org.codehaus.groovy.control.customizers.ASTTransformationCustomizer; +import org.codehaus.groovy.control.customizers.ImportCustomizer; +import org.codehaus.groovy.control.customizers.SecureASTCustomizer; + +import static java.util.Collections.singletonList; +import static java.util.Collections.singletonMap; + +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +public class DSL { + private final DelegatingScript script; + + private final FilterSpec filterSpec; + + public static DSL of(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig config, + final String dsl) throws ModuleStartException { + final CompilerConfiguration cc = new CompilerConfiguration(); + final ASTTransformationCustomizer customizer = + new ASTTransformationCustomizer( + singletonMap( + "extensions", + singletonList(LALPrecompiledExtension.class.getName()) + ), + CompileStatic.class + ); + cc.addCompilationCustomizers(customizer); + final SecureASTCustomizer secureASTCustomizer = new SecureASTCustomizer(); + secureASTCustomizer.setDisallowedStatements( + ImmutableList.>builder() + .add(WhileStatement.class) + .add(DoWhileStatement.class) + .add(ForStatement.class) + .build()); + // noinspection rawtypes + secureASTCustomizer.setAllowedReceiversClasses( + ImmutableList.builder() + .add(Object.class) + .add(Map.class) + .add(List.class) + .add(Array.class) + .add(GString.class) + .add(String.class) + .add(ProcessRegistry.class) + .build()); + cc.addCompilationCustomizers(secureASTCustomizer); + cc.setScriptBaseClass(LALDelegatingScript.class.getName()); + + ImportCustomizer icz = new ImportCustomizer(); + icz.addImport("ProcessRegistry", ProcessRegistry.class.getName()); + cc.addCompilationCustomizers(icz); + + final GroovyShell sh = new GroovyShell(cc); + final DelegatingScript script = (DelegatingScript) sh.parse(dsl); + final FilterSpec filterSpec = new FilterSpec(moduleManager, config); + script.setDelegate(filterSpec); + + return new DSL(script, filterSpec); + } + + public void bind(final Binding binding) { + this.filterSpec.bind(binding); + } + + public void evaluate() { + script.run(); + } +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java similarity index 100% rename from oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java rename to test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java new file mode 100644 index 000000000000..f96b02f485a2 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl; + +import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; + +/** + * Pure Java replacement for Groovy-based LAL DelegatingScript. + * Each transpiled LAL expression implements this interface. + */ +@FunctionalInterface +public interface LalExpression { + void execute(FilterSpec filterSpec, Binding binding); +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java new file mode 100644 index 000000000000..d815095776dc --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec; + +import groovy.lang.Closure; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.experimental.Accessors; +import org.apache.skywalking.apm.network.common.v3.KeyStringValuePair; +import org.apache.skywalking.oap.log.analyzer.dsl.Binding; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +@Getter +@RequiredArgsConstructor +@Accessors(fluent = true) +public abstract class AbstractSpec { + private final ModuleManager moduleManager; + + private final LogAnalyzerModuleConfig moduleConfig; + + protected static final ThreadLocal BINDING = ThreadLocal.withInitial(Binding::new); + + public void bind(final Binding b) { + BINDING.set(b); + } + + @SuppressWarnings("unused") + public void abort(final Closure cl) { + BINDING.get().abort(); + } + + @SuppressWarnings("unused") + public Object propertyMissing(final String name) { + return BINDING.get().getVariable(name); + } + + @SuppressWarnings("unused") + public String tag(String key) { + return BINDING.get().log().getTags().getDataList() + .stream() + .filter(data -> key.equals(data.getKey())) + .map(KeyStringValuePair::getValue) + .findFirst() + .orElse(""); + } +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java similarity index 100% rename from oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java rename to test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java new file mode 100644 index 000000000000..2a51d10d4f1b --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java @@ -0,0 +1,443 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor; + +import com.google.common.base.Joiner; +import com.google.common.collect.ImmutableMap; +import groovy.lang.Closure; +import groovy.lang.DelegatesTo; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.function.Consumer; +import java.util.stream.Collectors; +import lombok.experimental.Delegate; +import org.apache.commons.lang3.StringUtils; +import org.apache.skywalking.apm.network.common.v3.KeyStringValuePair; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.apm.network.logging.v3.TraceContext; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.sampledtrace.SampledTraceSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.slowsql.SlowSqlSpec; +import org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleProvider; +import org.apache.skywalking.oap.meter.analyzer.MetricConvert; +import org.apache.skywalking.oap.meter.analyzer.dsl.Sample; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyBuilder; +import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.DatabaseSlowStatementBuilder; +import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.SampledTraceBuilder; +import org.apache.skywalking.oap.server.core.CoreModule; +import org.apache.skywalking.oap.server.core.analysis.DownSampling; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.TimeBucket; +import org.apache.skywalking.oap.server.core.analysis.record.Record; +import org.apache.skywalking.oap.server.core.analysis.worker.RecordStreamProcessor; +import org.apache.skywalking.oap.server.core.config.NamingControl; +import org.apache.skywalking.oap.server.core.source.ISource; +import org.apache.skywalking.oap.server.core.source.ServiceMeta; +import org.apache.skywalking.oap.server.core.source.SourceReceiver; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.apache.skywalking.oap.server.library.util.CollectionUtils; +import org.apache.skywalking.oap.server.library.util.StringUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static java.util.Objects.nonNull; +import static org.apache.skywalking.oap.server.library.util.StringUtil.isNotBlank; + +public class ExtractorSpec extends AbstractSpec { + private static final Logger LOGGER = LoggerFactory.getLogger(SlowSqlSpec.class); + + private final List metricConverts; + + private final SlowSqlSpec slowSql; + private final SampledTraceSpec sampledTrace; + + private final NamingControl namingControl; + + private final SourceReceiver sourceReceiver; + + public ExtractorSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) throws ModuleStartException { + super(moduleManager, moduleConfig); + + LogAnalyzerModuleProvider provider = (LogAnalyzerModuleProvider) moduleManager + .find(LogAnalyzerModule.NAME).provider(); + + metricConverts = provider.getMetricConverts(); + + slowSql = new SlowSqlSpec(moduleManager(), moduleConfig()); + sampledTrace = new SampledTraceSpec(moduleManager(), moduleConfig()); + + namingControl = moduleManager.find(CoreModule.NAME) + .provider() + .getService(NamingControl.class); + + sourceReceiver = moduleManager.find(CoreModule.NAME).provider().getService(SourceReceiver.class); + } + + @SuppressWarnings("unused") + public void service(final String service) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(service)) { + BINDING.get().log().setService(service); + } + } + + @SuppressWarnings("unused") + public void instance(final String instance) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(instance)) { + BINDING.get().log().setServiceInstance(instance); + } + } + + @SuppressWarnings("unused") + public void endpoint(final String endpoint) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(endpoint)) { + BINDING.get().log().setEndpoint(endpoint); + } + } + + @SuppressWarnings("unused") + public void tag(final Map kv) { + if (BINDING.get().shouldAbort()) { + return; + } + if (CollectionUtils.isEmpty(kv)) { + return; + } + final LogData.Builder logData = BINDING.get().log(); + logData.setTags( + logData.getTags() + .toBuilder() + .addAllData( + kv.entrySet() + .stream() + .filter(it -> isNotBlank(it.getKey())) + .filter(it -> nonNull(it.getValue()) && + isNotBlank(Objects.toString(it.getValue()))) + .map(it -> { + final Object val = it.getValue(); + String valStr = Objects.toString(val); + if (Collection.class.isAssignableFrom(val.getClass())) { + valStr = Joiner.on(",").skipNulls().join((Collection) val); + } + return KeyStringValuePair.newBuilder() + .setKey(it.getKey()) + .setValue(valStr) + .build(); + }) + .collect(Collectors.toList()) + ) + ); + } + + @SuppressWarnings("unused") + public void traceId(final String traceId) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(traceId)) { + final LogData.Builder logData = BINDING.get().log(); + final TraceContext.Builder traceContext = logData.getTraceContext().toBuilder(); + traceContext.setTraceId(traceId); + logData.setTraceContext(traceContext); + } + } + + @SuppressWarnings("unused") + public void segmentId(final String segmentId) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(segmentId)) { + final LogData.Builder logData = BINDING.get().log(); + final TraceContext.Builder traceContext = logData.getTraceContext().toBuilder(); + traceContext.setTraceSegmentId(segmentId); + logData.setTraceContext(traceContext); + } + } + + @SuppressWarnings("unused") + public void spanId(final String spanId) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(spanId)) { + final LogData.Builder logData = BINDING.get().log(); + final TraceContext.Builder traceContext = logData.getTraceContext().toBuilder(); + traceContext.setSpanId(Integer.parseInt(spanId)); + logData.setTraceContext(traceContext); + } + } + + @SuppressWarnings("unused") + public void timestamp(final String timestamp) { + timestamp(timestamp, null); + } + + @SuppressWarnings("unused") + public void timestamp(final String timestamp, final String formatPattern) { + if (BINDING.get().shouldAbort()) { + return; + } + if (StringUtil.isEmpty(timestamp)) { + return; + } + + if (StringUtil.isEmpty(formatPattern)) { + if (StringUtils.isNumeric(timestamp)) { + BINDING.get().log().setTimestamp(Long.parseLong(timestamp)); + } + } else { + SimpleDateFormat format = new SimpleDateFormat(formatPattern); + try { + BINDING.get().log().setTimestamp(format.parse(timestamp).getTime()); + } catch (ParseException e) { + // ignore + } + } + } + + @SuppressWarnings("unused") + public void layer(final String layer) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(layer)) { + final LogData.Builder logData = BINDING.get().log(); + logData.setLayer(layer); + } + } + + @SuppressWarnings("unused") + public void metrics(@DelegatesTo(SampleBuilder.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + final SampleBuilder builder = new SampleBuilder(); + cl.setDelegate(builder); + cl.call(); + + final Sample sample = builder.build(); + final SampleFamily sampleFamily = SampleFamilyBuilder.newBuilder(sample).build(); + + final Optional> possibleMetricsContainer = BINDING.get().metricsContainer(); + + if (possibleMetricsContainer.isPresent()) { + possibleMetricsContainer.get().add(sampleFamily); + } else { + metricConverts.forEach(it -> it.toMeter( + ImmutableMap.builder() + .put(sample.getName(), sampleFamily) + .build() + )); + } + } + + @SuppressWarnings("unused") + public void slowSql(@DelegatesTo(SlowSqlSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + LogData.Builder log = BINDING.get().log(); + if (log.getLayer() == null + || log.getService() == null + || log.getTimestamp() < 1) { + LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); + return; + } + DatabaseSlowStatementBuilder builder = new DatabaseSlowStatementBuilder(namingControl); + builder.setLayer(Layer.nameOf(log.getLayer())); + + builder.setServiceName(log.getService()); + + BINDING.get().databaseSlowStatement(builder); + + cl.setDelegate(slowSql); + cl.call(); + + if (builder.getId() == null + || builder.getLatency() < 1 + || builder.getStatement() == null) { + LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); + return; + } + + long timeBucketForDB = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Second); + builder.setTimeBucket(timeBucketForDB); + builder.setTimestamp(log.getTimestamp()); + + builder.prepare(); + sourceReceiver.receive(builder.toDatabaseSlowStatement()); + + ServiceMeta serviceMeta = new ServiceMeta(); + serviceMeta.setName(builder.getServiceName()); + serviceMeta.setLayer(builder.getLayer()); + long timeBucket = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Minute); + serviceMeta.setTimeBucket(timeBucket); + sourceReceiver.receive(serviceMeta); + } + + @SuppressWarnings("unused") + public void sampledTrace(@DelegatesTo(SampledTraceSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + LogData.Builder log = BINDING.get().log(); + SampledTraceBuilder builder = new SampledTraceBuilder(namingControl); + builder.setLayer(log.getLayer()); + builder.setTimestamp(log.getTimestamp()); + builder.setServiceName(log.getService()); + builder.setServiceInstanceName(log.getServiceInstance()); + builder.setTraceId(log.getTraceContext().getTraceId()); + BINDING.get().sampledTrace(builder); + + cl.setDelegate(sampledTrace); + cl.call(); + + builder.validate(); + final Record record = builder.toRecord(); + final ISource entity = builder.toEntity(); + RecordStreamProcessor.getInstance().in(record); + sourceReceiver.receive(entity); + } + + public void metrics(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + final SampleBuilder builder = new SampleBuilder(); + consumer.accept(builder); + + final Sample sample = builder.build(); + final SampleFamily sampleFamily = SampleFamilyBuilder.newBuilder(sample).build(); + + final Optional> possibleMetricsContainer = BINDING.get().metricsContainer(); + + if (possibleMetricsContainer.isPresent()) { + possibleMetricsContainer.get().add(sampleFamily); + } else { + metricConverts.forEach(it -> it.toMeter( + ImmutableMap.builder() + .put(sample.getName(), sampleFamily) + .build() + )); + } + } + + public void slowSql(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + LogData.Builder log = BINDING.get().log(); + if (log.getLayer() == null + || log.getService() == null + || log.getTimestamp() < 1) { + LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); + return; + } + DatabaseSlowStatementBuilder builder = new DatabaseSlowStatementBuilder(namingControl); + builder.setLayer(Layer.nameOf(log.getLayer())); + + builder.setServiceName(log.getService()); + + BINDING.get().databaseSlowStatement(builder); + + consumer.accept(slowSql); + + if (builder.getId() == null + || builder.getLatency() < 1 + || builder.getStatement() == null) { + LOGGER.warn("SlowSql extracts failed, maybe something is not configured."); + return; + } + + long timeBucketForDB = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Second); + builder.setTimeBucket(timeBucketForDB); + builder.setTimestamp(log.getTimestamp()); + + builder.prepare(); + sourceReceiver.receive(builder.toDatabaseSlowStatement()); + + ServiceMeta serviceMeta = new ServiceMeta(); + serviceMeta.setName(builder.getServiceName()); + serviceMeta.setLayer(builder.getLayer()); + long timeBucket = TimeBucket.getTimeBucket(log.getTimestamp(), DownSampling.Minute); + serviceMeta.setTimeBucket(timeBucket); + sourceReceiver.receive(serviceMeta); + } + + public void sampledTrace(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + LogData.Builder log = BINDING.get().log(); + SampledTraceBuilder builder = new SampledTraceBuilder(namingControl); + builder.setLayer(log.getLayer()); + builder.setTimestamp(log.getTimestamp()); + builder.setServiceName(log.getService()); + builder.setServiceInstanceName(log.getServiceInstance()); + builder.setTraceId(log.getTraceContext().getTraceId()); + BINDING.get().sampledTrace(builder); + + consumer.accept(sampledTrace); + + builder.validate(); + final Record record = builder.toRecord(); + final ISource entity = builder.toEntity(); + RecordStreamProcessor.getInstance().in(record); + sourceReceiver.receive(entity); + } + + public static class SampleBuilder { + @Delegate + private final Sample.SampleBuilder sampleBuilder = Sample.builder(); + + @SuppressWarnings("unused") + public Sample.SampleBuilder labels(final Map labels) { + final Map filtered = + labels.entrySet() + .stream() + .filter(it -> isNotBlank(it.getKey()) && nonNull(it.getValue())) + .collect( + Collectors.toMap( + Map.Entry::getKey, + it -> Objects.toString(it.getValue()) + ) + ); + return sampleBuilder.labels(ImmutableMap.copyOf(filtered)); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java new file mode 100644 index 000000000000..2034548bf64c --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.sampledtrace; + +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.SampledTraceBuilder; +import org.apache.skywalking.oap.server.core.source.DetectPoint; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +import static java.util.Objects.nonNull; + +public class SampledTraceSpec extends AbstractSpec { + public SampledTraceSpec(ModuleManager moduleManager, LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + } + + public void latency(final Long latency) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(latency)) { + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setLatency(latency); + } + } + + public void uri(final String uri) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(uri)) { + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setUri(uri); + } + } + + public void reason(final String reason) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(reason)) { + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setReason(SampledTraceBuilder.Reason.valueOf(reason.toUpperCase())); + } + } + + public void processId(final String id) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(id)) { + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setProcessId(id); + } + } + + public void destProcessId(final String id) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(id)) { + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setDestProcessId(id); + } + } + + public void detectPoint(String detectPoint) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(detectPoint)) { + final DetectPoint point = DetectPoint.valueOf(detectPoint.toUpperCase()); + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setDetectPoint(point); + } + } + + public void componentId(final int id) { + if (BINDING.get().shouldAbort()) { + return; + } + if (id > 0) { + final SampledTraceBuilder sampledTraceBuilder = BINDING.get().sampledTraceBuilder(); + sampledTraceBuilder.setComponentId(id); + } + } + +} \ No newline at end of file diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java new file mode 100644 index 000000000000..5230352528b8 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.slowsql; + +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; + +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.analyzer.provider.trace.parser.listener.DatabaseSlowStatementBuilder; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +import static java.util.Objects.nonNull; + +public class SlowSqlSpec extends AbstractSpec { + + public SlowSqlSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + } + + public void latency(final Long latency) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(latency)) { + final DatabaseSlowStatementBuilder databaseSlowStatementBuilder = BINDING.get().databaseSlowStatement(); + databaseSlowStatementBuilder.setLatency(latency); + } + } + + public void statement(final String statement) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(statement)) { + final DatabaseSlowStatementBuilder databaseSlowStatementBuilder = BINDING.get().databaseSlowStatement(); + databaseSlowStatementBuilder.setStatement(statement); + } + } + + public void id(final String id) { + if (BINDING.get().shouldAbort()) { + return; + } + if (nonNull(id)) { + final DatabaseSlowStatementBuilder databaseSlowStatementBuilder = BINDING.get().databaseSlowStatement(); + databaseSlowStatementBuilder.setId(id); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java new file mode 100644 index 000000000000..b83e71b8b849 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java @@ -0,0 +1,358 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.filter; + +import com.fasterxml.jackson.core.type.TypeReference; +import com.google.protobuf.Message; +import com.google.protobuf.TextFormat; +import groovy.lang.Closure; +import groovy.lang.DelegatesTo; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; + +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.log.analyzer.dsl.Binding; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.ExtractorSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.parser.JsonParserSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.parser.TextParserSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.parser.YamlParserSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.SinkSpec; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogSinkListenerFactory; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.RecordSinkListener; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.TrafficSinkListener; +import org.apache.skywalking.oap.server.core.source.Log; + +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class FilterSpec extends AbstractSpec { + private static final Logger LOGGER = LoggerFactory.getLogger(FilterSpec.class); + + private final List sinkListenerFactories; + + private final TextParserSpec textParser; + + private final JsonParserSpec jsonParser; + + private final YamlParserSpec yamlParser; + + private final ExtractorSpec extractor; + + private final SinkSpec sink; + + private final TypeReference> parsedType; + + public FilterSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) throws ModuleStartException { + super(moduleManager, moduleConfig); + + parsedType = new TypeReference>() { + }; + + sinkListenerFactories = Arrays.asList( + new RecordSinkListener.Factory(moduleManager(), moduleConfig()), + new TrafficSinkListener.Factory(moduleManager(), moduleConfig()) + ); + + textParser = new TextParserSpec(moduleManager(), moduleConfig()); + jsonParser = new JsonParserSpec(moduleManager(), moduleConfig()); + yamlParser = new YamlParserSpec(moduleManager(), moduleConfig()); + + extractor = new ExtractorSpec(moduleManager(), moduleConfig()); + + sink = new SinkSpec(moduleManager(), moduleConfig()); + } + + @SuppressWarnings("unused") + public void text(@DelegatesTo(TextParserSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + cl.setDelegate(textParser); + cl.call(); + } + + @SuppressWarnings("unused") + public void json(@DelegatesTo(JsonParserSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + cl.setDelegate(jsonParser); + cl.call(); + + final LogData.Builder logData = BINDING.get().log(); + try { + + final Map parsed = jsonParser.create().readValue( + logData.getBody().getJson().getJson(), parsedType + ); + + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (jsonParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + @SuppressWarnings({"unused"}) + public void yaml(@DelegatesTo(YamlParserSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + cl.setDelegate(yamlParser); + cl.call(); + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = yamlParser.create().load( + logData.getBody().getYaml().getYaml() + ); + + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (yamlParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + @SuppressWarnings("unused") + public void extractor(@DelegatesTo(ExtractorSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + cl.setDelegate(extractor); + cl.call(); + } + + @SuppressWarnings("unused") + public void sink(@DelegatesTo(SinkSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + cl.setDelegate(sink); + cl.call(); + + final Binding b = BINDING.get(); + final LogData.Builder logData = b.log(); + final Message extraLog = b.extraLog(); + + if (!b.shouldSave()) { + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Log is dropped: {}", TextFormat.shortDebugString(logData)); + } + return; + } + + final Optional> container = BINDING.get().logContainer(); + if (container.isPresent()) { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .filter(it -> it instanceof RecordSinkListener) + .map(it -> it.parse(logData, extraLog)) + .map(it -> (RecordSinkListener) it) + .map(RecordSinkListener::getLog) + .findFirst() + .ifPresent(log -> container.get().set(log)); + } else { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .forEach(it -> it.parse(logData, extraLog).build()); + } + } + + @SuppressWarnings("unused") + public void filter(final Closure cl) { + cl.call(); + } + + public void text(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(textParser); + } + + public void text() { + if (BINDING.get().shouldAbort()) { + return; + } + } + + public void json(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(jsonParser); + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = jsonParser.create().readValue( + logData.getBody().getJson().getJson(), parsedType + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (jsonParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void json() { + if (BINDING.get().shouldAbort()) { + return; + } + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = jsonParser.create().readValue( + logData.getBody().getJson().getJson(), parsedType + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (jsonParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void yaml(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(yamlParser); + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = yamlParser.create().load( + logData.getBody().getYaml().getYaml() + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (yamlParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void yaml() { + if (BINDING.get().shouldAbort()) { + return; + } + + final LogData.Builder logData = BINDING.get().log(); + try { + final Map parsed = yamlParser.create().load( + logData.getBody().getYaml().getYaml() + ); + BINDING.get().parsed(parsed); + } catch (final Exception e) { + if (yamlParser.abortOnFailure()) { + BINDING.get().abort(); + } + } + } + + public void extractor(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(extractor); + } + + public void sink(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(sink); + + final Binding b = BINDING.get(); + final LogData.Builder logData = b.log(); + final Message extraLog = b.extraLog(); + + if (!b.shouldSave()) { + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Log is dropped: {}", TextFormat.shortDebugString(logData)); + } + return; + } + + final Optional> container = BINDING.get().logContainer(); + if (container.isPresent()) { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .filter(it -> it instanceof RecordSinkListener) + .map(it -> it.parse(logData, extraLog)) + .map(it -> (RecordSinkListener) it) + .map(RecordSinkListener::getLog) + .findFirst() + .ifPresent(log -> container.get().set(log)); + } else { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .forEach(it -> it.parse(logData, extraLog).build()); + } + } + + public void sink() { + if (BINDING.get().shouldAbort()) { + return; + } + + final Binding b = BINDING.get(); + final LogData.Builder logData = b.log(); + final Message extraLog = b.extraLog(); + + if (!b.shouldSave()) { + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Log is dropped: {}", TextFormat.shortDebugString(logData)); + } + return; + } + + final Optional> container = BINDING.get().logContainer(); + if (container.isPresent()) { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .filter(it -> it instanceof RecordSinkListener) + .map(it -> it.parse(logData, extraLog)) + .map(it -> (RecordSinkListener) it) + .map(RecordSinkListener::getLog) + .findFirst() + .ifPresent(log -> container.get().set(log)); + } else { + sinkListenerFactories.stream() + .map(LogSinkListenerFactory::create) + .forEach(it -> it.parse(logData, extraLog).build()); + } + } + + public void abort() { + BINDING.get().abort(); + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java new file mode 100644 index 000000000000..a7db11902734 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.parser; + +import lombok.experimental.Accessors; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +@Accessors +public class AbstractParserSpec extends AbstractSpec { + /** + * Whether the filter chain should abort when parsing the logs failed. + * + * Failing to parse the logs means either parsing throws exceptions or the logs not matching the + * desired patterns. + */ + private boolean abortOnFailure = true; + + public AbstractParserSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + } + + @SuppressWarnings("unused") // used in user LAL scripts + public void abortOnFailure(final boolean abortOnFailure) { + this.abortOnFailure = abortOnFailure; + } + + public boolean abortOnFailure() { + return this.abortOnFailure; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java new file mode 100644 index 000000000000..1fabdbcf1c86 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.parser; + +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +public class JsonParserSpec extends AbstractParserSpec { + private final ObjectMapper mapper; + + public JsonParserSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + + // We just create a mapper instance in advance for now (for the sake of performance), + // when we want to provide some extra options, we'll move this into method "create" then. + mapper = new ObjectMapper(); + } + + public ObjectMapper create() { + return mapper; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java new file mode 100644 index 000000000000..f20a1e9d810b --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java @@ -0,0 +1,57 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.parser; + +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +public class TextParserSpec extends AbstractParserSpec { + public TextParserSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + } + + @SuppressWarnings("unused") + public void regexp(final String regexp) { + regexp(Pattern.compile(regexp)); + } + + public void regexp(final Pattern pattern) { + if (BINDING.get().shouldAbort()) { + return; + } + final LogData.Builder log = BINDING.get().log(); + final Matcher matcher = pattern.matcher(log.getBody().getText().getText()); + final boolean matched = matcher.find(); + if (matched) { + BINDING.get().parsed(matcher); + } else if (abortOnFailure()) { + BINDING.get().abort(); + } + } + + public boolean grok(final String grok) { + // TODO + return false; + } + +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java new file mode 100644 index 000000000000..2b99b30bed85 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.parser; + +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.yaml.snakeyaml.DumperOptions; +import org.yaml.snakeyaml.LoaderOptions; +import org.yaml.snakeyaml.Yaml; +import org.yaml.snakeyaml.constructor.SafeConstructor; +import org.yaml.snakeyaml.representer.Representer; + +public class YamlParserSpec extends AbstractParserSpec { + private final LoaderOptions loaderOptions; + + public YamlParserSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + + loaderOptions = new LoaderOptions(); + } + + public Yaml create() { + final var dumperOptions = new DumperOptions(); + return new Yaml( + new SafeConstructor(loaderOptions), + new Representer(dumperOptions), + dumperOptions, loaderOptions); + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java new file mode 100644 index 000000000000..f10d471ee184 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink; + +import groovy.lang.Closure; +import groovy.lang.DelegatesTo; +import groovy.lang.GString; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Consumer; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.PossibilitySampler; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.RateLimitingSampler; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.Sampler; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +public class SamplerSpec extends AbstractSpec { + private final Map rateLimitSamplers; + private final Map rateLimitSamplersByString; + private final Map possibilitySamplers; + private final RateLimitingSampler.ResetHandler rlsResetHandler; + + public SamplerSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + + rateLimitSamplers = new ConcurrentHashMap<>(); + rateLimitSamplersByString = new ConcurrentHashMap<>(); + possibilitySamplers = new ConcurrentHashMap<>(); + rlsResetHandler = new RateLimitingSampler.ResetHandler(); + } + + @SuppressWarnings("unused") + public void rateLimit(final GString id, @DelegatesTo(RateLimitingSampler.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + + final Sampler sampler = rateLimitSamplers.computeIfAbsent(id, $ -> new RateLimitingSampler(rlsResetHandler).start()); + + cl.setDelegate(sampler); + cl.call(); + + sampleWith(sampler); + } + + @SuppressWarnings("unused") + public void rateLimit(final String id, final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + + final Sampler sampler = rateLimitSamplersByString.computeIfAbsent( + id, $ -> new RateLimitingSampler(rlsResetHandler).start()); + + consumer.accept((RateLimitingSampler) sampler); + + sampleWith(sampler); + } + + @SuppressWarnings("unused") + public void possibility(final int percentage, @DelegatesTo(PossibilitySampler.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + + final Sampler sampler = possibilitySamplers.computeIfAbsent(percentage, $ -> new PossibilitySampler(percentage).start()); + + cl.setDelegate(sampler); + cl.call(); + + sampleWith(sampler); + } + + private void sampleWith(final Sampler sampler) { + if (BINDING.get().shouldAbort()) { + return; + } + if (sampler.sample()) { + BINDING.get().save(); + } else { + BINDING.get().drop(); + } + } + +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java new file mode 100644 index 000000000000..f2ae371a21e2 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java @@ -0,0 +1,84 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink; + +import groovy.lang.Closure; +import groovy.lang.DelegatesTo; +import java.util.function.Consumer; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +public class SinkSpec extends AbstractSpec { + + private final SamplerSpec sampler; + + public SinkSpec(final ModuleManager moduleManager, + final LogAnalyzerModuleConfig moduleConfig) { + super(moduleManager, moduleConfig); + + sampler = new SamplerSpec(moduleManager(), moduleConfig()); + } + + @SuppressWarnings("unused") + public void sampler(@DelegatesTo(SamplerSpec.class) final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + cl.setDelegate(sampler); + cl.call(); + } + + public void sampler(final Consumer consumer) { + if (BINDING.get().shouldAbort()) { + return; + } + consumer.accept(sampler); + } + + @SuppressWarnings("unused") + public void enforcer(final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + BINDING.get().save(); + } + + public void enforcer() { + if (BINDING.get().shouldAbort()) { + return; + } + BINDING.get().save(); + } + + @SuppressWarnings("unused") + public void dropper(final Closure cl) { + if (BINDING.get().shouldAbort()) { + return; + } + BINDING.get().drop(); + } + + public void dropper() { + if (BINDING.get().shouldAbort()) { + return; + } + BINDING.get().drop(); + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java new file mode 100644 index 000000000000..aeb1426a316e --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler; + +import io.netty.util.internal.ThreadLocalRandom; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.experimental.Accessors; + +@RequiredArgsConstructor +@Accessors(fluent = true) +@EqualsAndHashCode(of = {"percentage"}) +public class PossibilitySampler implements Sampler { + @Getter + private final int percentage; + + private final ThreadLocalRandom random = ThreadLocalRandom.current(); + + @Override + public PossibilitySampler start() { + return this; + } + + @Override + public void close() { + } + + @Override + public boolean sample() { + return random.nextInt(100) < percentage; + } + + @Override + public PossibilitySampler reset() { + return this; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java new file mode 100644 index 000000000000..3f1c548e3d04 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler; + +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.Setter; +import lombok.experimental.Accessors; +import lombok.extern.slf4j.Slf4j; + +@Accessors(fluent = true) +@EqualsAndHashCode(of = {"rpm"}) +public class RateLimitingSampler implements Sampler { + @Getter + @Setter + private volatile int rpm; + + private final AtomicInteger factor = new AtomicInteger(); + + private final ResetHandler resetHandler; + + public RateLimitingSampler(final ResetHandler resetHandler) { + this.resetHandler = resetHandler; + } + + @Override + public RateLimitingSampler start() { + resetHandler.start(this); + return this; + } + + @Override + public void close() { + resetHandler.close(this); + } + + @Override + public boolean sample() { + return factor.getAndIncrement() < rpm; + } + + @Override + public RateLimitingSampler reset() { + factor.set(0); + return this; + } + + @Slf4j + public static class ResetHandler { + private final List samplers = new ArrayList<>(); + + private volatile ScheduledFuture future; + + private volatile boolean started = false; + + private synchronized void start(final Sampler sampler) { + samplers.add(sampler); + + if (!started) { + future = Executors.newSingleThreadScheduledExecutor() + .scheduleAtFixedRate(this::reset, 1, 1, TimeUnit.MINUTES); + started = true; + } + } + + private synchronized void close(final Sampler sampler) { + samplers.remove(sampler); + + if (samplers.isEmpty() && future != null) { + future.cancel(true); + started = false; + } + } + + private synchronized void reset() { + samplers.forEach(sampler -> { + try { + sampler.reset(); + } catch (final Exception e) { + log.error("Failed to reset sampler {}.", sampler, e); + } + }); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java new file mode 100644 index 000000000000..61e315c29a2c --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler; + +public interface Sampler extends AutoCloseable { + Sampler NOOP = new Sampler() { + @Override + public boolean sample() { + return false; + } + + @Override + public void close() { + } + }; + + boolean sample(); + + default Sampler start() { + return this; + } + + default Sampler reset() { + return this; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java new file mode 100644 index 000000000000..6b9ecefc0f6a --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.module; + +import org.apache.skywalking.oap.log.analyzer.provider.log.ILogAnalyzerService; +import org.apache.skywalking.oap.server.library.module.ModuleDefine; + +public class LogAnalyzerModule extends ModuleDefine { + public static final String NAME = "log-analyzer"; + + public LogAnalyzerModule() { + super(NAME); + } + + @Override + public Class[] services() { + return new Class[] { + ILogAnalyzerService.class + }; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java new file mode 100644 index 000000000000..82323384c8e8 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.provider; + +import lombok.Data; + +@Data +public class LALConfig { + private String name; + + private String dsl; + + private String layer; +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java new file mode 100644 index 000000000000..4bf81310bff8 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.provider; + +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileReader; +import java.io.IOException; +import java.io.Reader; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Objects; +import java.util.stream.Collectors; +import lombok.Data; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.apache.skywalking.oap.server.library.util.ResourceUtils; +import org.yaml.snakeyaml.Yaml; + +import static com.google.common.base.Preconditions.checkArgument; +import static com.google.common.io.Files.getNameWithoutExtension; +import static org.apache.skywalking.oap.server.library.util.StringUtil.isNotBlank; +import static org.apache.skywalking.oap.server.library.util.CollectionUtils.isEmpty; + +@Data +@Slf4j +public class LALConfigs { + private List rules; + + public static List load(final String path, final List files) throws Exception { + if (isEmpty(files)) { + return Collections.emptyList(); + } + + checkArgument(isNotBlank(path), "path cannot be blank"); + + try { + final File[] rules = ResourceUtils.getPathFiles(path); + + return Arrays.stream(rules) + .filter(File::isFile) + .filter(it -> { + //noinspection UnstableApiUsage + return files.contains(getNameWithoutExtension(it.getName())); + }) + .map(f -> { + try (final Reader r = new FileReader(f)) { + return new Yaml().loadAs(r, LALConfigs.class); + } catch (IOException e) { + log.debug("Failed to read file {}", f, e); + } + return null; + }) + .filter(Objects::nonNull) + .collect(Collectors.toList()); + } catch (FileNotFoundException e) { + throw new ModuleStartException("Failed to load LAL config rules", e); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java new file mode 100644 index 000000000000..de995c51b8e0 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider; + +import com.google.common.base.Splitter; +import com.google.common.base.Strings; + +import java.io.IOException; +import java.util.List; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.Setter; +import org.apache.skywalking.oap.meter.analyzer.prometheus.rule.Rule; +import org.apache.skywalking.oap.meter.analyzer.prometheus.rule.Rules; +import org.apache.skywalking.oap.server.library.module.ModuleConfig; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; + +import static java.util.Objects.nonNull; + +@EqualsAndHashCode(callSuper = false) +public class LogAnalyzerModuleConfig extends ModuleConfig { + @Getter + @Setter + private String lalPath = "lal"; + + @Getter + @Setter + private String malPath = "log-mal-rules"; + + @Getter + @Setter + private String lalFiles = "default.yaml"; + + @Getter + @Setter + private String malFiles; + + private List meterConfigs; + + public List lalFiles() { + return Splitter.on(",").omitEmptyStrings().trimResults().splitToList(Strings.nullToEmpty(getLalFiles())); + } + + public List malConfigs() throws ModuleStartException { + if (nonNull(meterConfigs)) { + return meterConfigs; + } + final List files = Splitter.on(",") + .omitEmptyStrings() + .splitToList(Strings.nullToEmpty(getMalFiles())); + try { + meterConfigs = Rules.loadRules(getMalPath(), files); + } catch (IOException e) { + throw new ModuleStartException("Failed to load MAL rules", e); + } + + return meterConfigs; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java new file mode 100644 index 000000000000..39a002168287 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java @@ -0,0 +1,102 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider; + +import java.util.List; +import java.util.stream.Collectors; +import lombok.Getter; +import org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule; +import org.apache.skywalking.oap.log.analyzer.provider.log.ILogAnalyzerService; +import org.apache.skywalking.oap.log.analyzer.provider.log.LogAnalyzerServiceImpl; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogFilterListener; +import org.apache.skywalking.oap.meter.analyzer.MetricConvert; +import org.apache.skywalking.oap.server.configuration.api.ConfigurationModule; +import org.apache.skywalking.oap.server.core.CoreModule; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem; +import org.apache.skywalking.oap.server.library.module.ModuleDefine; +import org.apache.skywalking.oap.server.library.module.ModuleProvider; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException; + +public class LogAnalyzerModuleProvider extends ModuleProvider { + @Getter + private LogAnalyzerModuleConfig moduleConfig; + + @Getter + private List metricConverts; + + private LogAnalyzerServiceImpl logAnalyzerService; + + @Override + public String name() { + return "default"; + } + + @Override + public Class module() { + return LogAnalyzerModule.class; + } + + @Override + public ConfigCreator newConfigCreator() { + return new ConfigCreator() { + @Override + public Class type() { + return LogAnalyzerModuleConfig.class; + } + + @Override + public void onInitialized(final LogAnalyzerModuleConfig initialized) { + moduleConfig = initialized; + } + }; + } + + @Override + public void prepare() throws ServiceNotProvidedException, ModuleStartException { + logAnalyzerService = new LogAnalyzerServiceImpl(getManager(), moduleConfig); + this.registerServiceImplementation(ILogAnalyzerService.class, logAnalyzerService); + } + + @Override + public void start() throws ServiceNotProvidedException, ModuleStartException { + MeterSystem meterSystem = getManager().find(CoreModule.NAME).provider().getService(MeterSystem.class); + metricConverts = moduleConfig.malConfigs() + .stream() + .map(it -> new MetricConvert(it, meterSystem)) + .collect(Collectors.toList()); + try { + logAnalyzerService.addListenerFactory(new LogFilterListener.Factory(getManager(), moduleConfig)); + } catch (final Exception e) { + throw new ModuleStartException("Failed to create LAL listener.", e); + } + } + + @Override + public void notifyAfterCompleted() throws ServiceNotProvidedException { + + } + + @Override + public String[] requiredModules() { + return new String[] { + CoreModule.NAME, + ConfigurationModule.NAME + }; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java new file mode 100644 index 000000000000..2b8b03ec69b4 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log; + +import java.util.List; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogAnalysisListenerFactory; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogSinkListenerFactory; + +public interface ILogAnalysisListenerManager { + + void addListenerFactory(LogAnalysisListenerFactory factory); + + List getLogAnalysisListenerFactories(); + + void addSinkListenerFactory(LogSinkListenerFactory factory); + + List getSinkListenerFactory(); +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java new file mode 100644 index 000000000000..653cf6dbb14e --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log; + +import com.google.protobuf.Message; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.server.library.module.Service; + +/** + * Analyze the collected log data. + */ +public interface ILogAnalyzerService extends Service { + + void doAnalysis(LogData.Builder log, Message extraLog); + + default void doAnalysis(LogData logData, Message extraLog) { + doAnalysis(logData.toBuilder(), extraLog); + } + +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java new file mode 100644 index 000000000000..73909b1813e2 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log; + +import com.google.protobuf.Message; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; + +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.server.core.UnexpectedException; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.library.util.StringUtil; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogAnalysisListener; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +/** + * Analyze the collected log data, is the entry point for log analysis. + */ +@Slf4j +@RequiredArgsConstructor +public class LogAnalyzer { + private final ModuleManager moduleManager; + private final LogAnalyzerModuleConfig moduleConfig; + private final ILogAnalysisListenerManager factoryManager; + + private final List listeners = new ArrayList<>(); + + public void doAnalysis(LogData.Builder builder, Message extraLog) { + if (StringUtil.isEmpty(builder.getService())) { + // If the service name is empty, the log will be ignored. + log.debug("The log is ignored because the Service name is empty"); + return; + } + Layer layer; + if ("".equals(builder.getLayer())) { + layer = Layer.GENERAL; + } else { + try { + layer = Layer.nameOf(builder.getLayer()); + } catch (UnexpectedException e) { + log.warn("The Layer {} is not found, abandon the log.", builder.getLayer()); + return; + } + } + + createAnalysisListeners(layer); + if (builder.getTimestamp() == 0) { + // If no timestamp, OAP server would use the received timestamp as log's timestamp + builder.setTimestamp(System.currentTimeMillis()); + } + + notifyAnalysisListener(builder, extraLog); + notifyAnalysisListenerToBuild(); + } + + private void notifyAnalysisListener(LogData.Builder builder, final Message extraLog) { + listeners.forEach(listener -> listener.parse(builder, extraLog)); + } + + private void notifyAnalysisListenerToBuild() { + listeners.forEach(LogAnalysisListener::build); + } + + private void createAnalysisListeners(Layer layer) { + factoryManager.getLogAnalysisListenerFactories() + .stream() + .map(factory -> factory.create(layer)) + .filter(Objects::nonNull) + .forEach(listeners::add); + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java new file mode 100644 index 000000000000..d5dbcd150ce3 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log; + +import com.google.protobuf.Message; +import java.util.ArrayList; +import java.util.List; +import lombok.RequiredArgsConstructor; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogAnalysisListenerFactory; +import org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogSinkListenerFactory; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +@RequiredArgsConstructor +public class LogAnalyzerServiceImpl implements ILogAnalyzerService, ILogAnalysisListenerManager { + private final ModuleManager moduleManager; + private final LogAnalyzerModuleConfig moduleConfig; + private final List analysisListenerFactories = new ArrayList<>(); + private final List sinkListenerFactories = new ArrayList<>(); + + @Override + public void doAnalysis(final LogData.Builder log, Message extraLog) { + LogAnalyzer analyzer = new LogAnalyzer(moduleManager, moduleConfig, this); + analyzer.doAnalysis(log, extraLog); + } + + @Override + public void addListenerFactory(final LogAnalysisListenerFactory factory) { + analysisListenerFactories.add(factory); + } + + @Override + public List getLogAnalysisListenerFactories() { + return analysisListenerFactories; + } + + @Override + public void addSinkListenerFactory(LogSinkListenerFactory factory) { + sinkListenerFactories.add(factory); + } + + @Override + public List getSinkListenerFactory() { + return sinkListenerFactories; + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java new file mode 100644 index 000000000000..13b610469ec5 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.analyzer; + +public class LogAnalyzerFactory { + +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java new file mode 100644 index 000000000000..f43c3a6e4c6a --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +import com.google.protobuf.Message; +import org.apache.skywalking.apm.network.logging.v3.LogData; + +/** + * LogAnalysisListener represents the callback when OAP does the log data analysis. + */ +public interface LogAnalysisListener { + /** + * The last step of the analysis process. Typically, the implementations execute corresponding DSL. + */ + void build(); + + /** + * Parse the raw data from the probe. + * @return {@code this} for chaining. + */ + LogAnalysisListener parse(LogData.Builder logData, final Message extraLog); +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java new file mode 100644 index 000000000000..8955adf2c1d3 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +import org.apache.skywalking.oap.server.core.analysis.Layer; + +/** + * LogAnalysisListenerFactory implementation creates the listener instance when required. + * Every LogAnalysisListener could have its own creation factory. + */ +public interface LogAnalysisListenerFactory { + + LogAnalysisListener create(Layer layer); +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java new file mode 100644 index 000000000000..8b9f0b1fb80c --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +import com.google.protobuf.Message; +import java.util.Collection; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; + +import java.util.HashMap; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.log.analyzer.dsl.Binding; +import org.apache.skywalking.oap.log.analyzer.dsl.DSL; +import org.apache.skywalking.oap.log.analyzer.provider.LALConfig; +import org.apache.skywalking.oap.log.analyzer.provider.LALConfigs; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; + +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleStartException; + +@Slf4j +@RequiredArgsConstructor +public class LogFilterListener implements LogAnalysisListener { + private final Collection dsls; + + @Override + public void build() { + dsls.forEach(dsl -> { + try { + dsl.evaluate(); + } catch (final Exception e) { + log.warn("Failed to evaluate dsl: {}", dsl, e); + } + }); + } + + @Override + public LogAnalysisListener parse(final LogData.Builder logData, + final Message extraLog) { + dsls.forEach(dsl -> dsl.bind(new Binding().log(logData.build()) + .extraLog(extraLog))); + return this; + } + + public static class Factory implements LogAnalysisListenerFactory { + private final Map> dsls; + + public Factory(final ModuleManager moduleManager, final LogAnalyzerModuleConfig config) throws Exception { + dsls = new HashMap<>(); + + final List configList = LALConfigs.load(config.getLalPath(), config.lalFiles()) + .stream() + .flatMap(it -> it.getRules().stream()) + .collect(Collectors.toList()); + for (final LALConfig c : configList) { + Layer layer = Layer.nameOf(c.getLayer()); + Map dsls = this.dsls.computeIfAbsent(layer, k -> new HashMap<>()); + if (dsls.put(c.getName(), DSL.of(moduleManager, config, c.getDsl())) != null) { + throw new ModuleStartException("Layer " + layer.name() + " has already set " + c.getName() + " rule."); + } + } + } + + @Override + public LogAnalysisListener create(Layer layer) { + if (layer == null) { + return null; + } + final Map dsl = dsls.get(layer); + if (dsl == null) { + return null; + } + return new LogFilterListener(dsl.values()); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java new file mode 100644 index 000000000000..0a763c040382 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +import com.google.protobuf.Message; +import org.apache.skywalking.apm.network.logging.v3.LogData; + +public interface LogSinkListener { + /** + * The last step of the sink process. Typically, the implementations forward the results to the source + * receiver. + */ + void build(); + + /** + * Parse the raw data from the probe. + * @return {@code this} for chaining. + */ + LogSinkListener parse(LogData.Builder logData, final Message extraLog); +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java new file mode 100644 index 000000000000..571bb843c2d4 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +/** + * LogSinkListenerFactory implementation creates the listener instance when required. + * Every LogSinkListener could have its own creation factory. + */ +public interface LogSinkListenerFactory { + LogSinkListener create(); +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java new file mode 100644 index 000000000000..0de97597603f --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java @@ -0,0 +1,178 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +import com.google.protobuf.Message; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashSet; +import java.util.List; +import java.util.UUID; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.SneakyThrows; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.apm.network.logging.v3.LogDataBody; +import org.apache.skywalking.apm.network.logging.v3.TraceContext; + +import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.TagType; +import org.apache.skywalking.oap.server.core.source.TagAutocomplete; +import org.apache.skywalking.oap.server.library.util.StringUtil; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.core.Const; +import org.apache.skywalking.oap.server.core.CoreModule; +import org.apache.skywalking.oap.server.core.analysis.IDManager; +import org.apache.skywalking.oap.server.core.analysis.TimeBucket; +import org.apache.skywalking.oap.server.core.analysis.manual.searchtag.Tag; +import org.apache.skywalking.oap.server.core.config.ConfigService; +import org.apache.skywalking.oap.server.core.config.NamingControl; +import org.apache.skywalking.oap.server.core.query.type.ContentType; +import org.apache.skywalking.oap.server.core.source.Log; +import org.apache.skywalking.oap.server.core.source.SourceReceiver; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.apache.skywalking.oap.server.library.util.ProtoBufJsonUtils.toJSON; + +/** + * RecordSinkListener forwards the log data to the persistence layer with the query required conditions. + */ +@RequiredArgsConstructor +public class RecordSinkListener implements LogSinkListener { + private static final Logger LOGGER = LoggerFactory.getLogger(RecordSinkListener.class); + private final SourceReceiver sourceReceiver; + private final NamingControl namingControl; + private final List searchableTagKeys; + @Getter + private final Log log = new Log(); + + @Override + public void build() { + sourceReceiver.receive(log); + addAutocompleteTags(); + } + + @Override + @SneakyThrows + public LogSinkListener parse(final LogData.Builder logData, + final Message extraLog) { + LogDataBody body = logData.getBody(); + log.setUniqueId(UUID.randomUUID().toString().replace("-", "")); + // timestamp + log.setTimestamp(logData.getTimestamp()); + log.setTimeBucket(TimeBucket.getRecordTimeBucket(logData.getTimestamp())); + + // service + String serviceName = namingControl.formatServiceName(logData.getService()); + String serviceId = IDManager.ServiceID.buildId(serviceName, true); + log.setServiceId(serviceId); + // service instance + if (StringUtil.isNotEmpty(logData.getServiceInstance())) { + log.setServiceInstanceId(IDManager.ServiceInstanceID.buildId( + serviceId, + namingControl.formatInstanceName(logData.getServiceInstance()) + )); + } + // endpoint + if (StringUtil.isNotEmpty(logData.getEndpoint())) { + String endpointName = namingControl.formatEndpointName(serviceName, logData.getEndpoint()); + log.setEndpointId(IDManager.EndpointID.buildId(serviceId, endpointName)); + } + // trace + TraceContext traceContext = logData.getTraceContext(); + if (StringUtil.isNotEmpty(traceContext.getTraceId())) { + log.setTraceId(traceContext.getTraceId()); + } + if (StringUtil.isNotEmpty(traceContext.getTraceSegmentId())) { + log.setTraceSegmentId(traceContext.getTraceSegmentId()); + log.setSpanId(traceContext.getSpanId()); + } + // content + if (body.hasText()) { + log.setContentType(ContentType.TEXT); + log.setContent(body.getText().getText()); + } else if (body.hasYaml()) { + log.setContentType(ContentType.YAML); + log.setContent(body.getYaml().getYaml()); + } else if (body.hasJson()) { + log.setContentType(ContentType.JSON); + log.setContent(body.getJson().getJson()); + } else if (extraLog != null) { + log.setContentType(ContentType.JSON); + log.setContent(toJSON(extraLog)); + } + if (logData.getTags().getDataCount() > 0) { + log.setTagsRawData(logData.getTags().toByteArray()); + } + log.getTags().addAll(appendSearchableTags(logData)); + return this; + } + + private Collection appendSearchableTags(LogData.Builder logData) { + HashSet logTags = new HashSet<>(); + logData.getTags().getDataList().forEach(tag -> { + if (searchableTagKeys.contains(tag.getKey())) { + final Tag logTag = new Tag(tag.getKey(), tag.getValue()); + if (tag.getValue().length() > Tag.TAG_LENGTH || logTag.toString().length() > Tag.TAG_LENGTH) { + if (LOGGER.isDebugEnabled()) { + LOGGER.debug("Log tag : {} length > : {}, dropped", logTag, Tag.TAG_LENGTH); + } + return; + } + logTags.add(logTag); + } + }); + return logTags; + } + + private void addAutocompleteTags() { + log.getTags().forEach(tag -> { + TagAutocomplete tagAutocomplete = new TagAutocomplete(); + tagAutocomplete.setTagKey(tag.getKey()); + tagAutocomplete.setTagValue(tag.getValue()); + tagAutocomplete.setTagType(TagType.LOG); + tagAutocomplete.setTimeBucket(TimeBucket.getMinuteTimeBucket(log.getTimestamp())); + sourceReceiver.receive(tagAutocomplete); + }); + } + + public static class Factory implements LogSinkListenerFactory { + private final SourceReceiver sourceReceiver; + private final NamingControl namingControl; + private final List searchableTagKeys; + + public Factory(ModuleManager moduleManager, LogAnalyzerModuleConfig moduleConfig) { + this.sourceReceiver = moduleManager.find(CoreModule.NAME) + .provider() + .getService(SourceReceiver.class); + this.namingControl = moduleManager.find(CoreModule.NAME) + .provider() + .getService(NamingControl.class); + ConfigService configService = moduleManager.find(CoreModule.NAME) + .provider() + .getService(ConfigService.class); + this.searchableTagKeys = Arrays.asList(configService.getSearchableLogsTags().split(Const.COMMA)); + } + + @Override + public RecordSinkListener create() { + return new RecordSinkListener(sourceReceiver, namingControl, searchableTagKeys); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java new file mode 100644 index 000000000000..1e7c9e3738e7 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java @@ -0,0 +1,118 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.provider.log.listener; + +import com.google.protobuf.Message; +import lombok.RequiredArgsConstructor; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.library.util.StringUtil; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.server.core.CoreModule; +import org.apache.skywalking.oap.server.core.analysis.DownSampling; +import org.apache.skywalking.oap.server.core.analysis.IDManager; +import org.apache.skywalking.oap.server.core.analysis.TimeBucket; +import org.apache.skywalking.oap.server.core.config.NamingControl; +import org.apache.skywalking.oap.server.core.source.EndpointMeta; +import org.apache.skywalking.oap.server.core.source.ServiceInstanceUpdate; +import org.apache.skywalking.oap.server.core.source.ServiceMeta; +import org.apache.skywalking.oap.server.core.source.SourceReceiver; +import org.apache.skywalking.oap.server.library.module.ModuleManager; + +import static java.util.Objects.nonNull; + +/** + * Generate service, service instance and endpoint traffic by log data. + */ +@RequiredArgsConstructor +public class TrafficSinkListener implements LogSinkListener { + private final SourceReceiver sourceReceiver; + private final NamingControl namingControl; + + private ServiceMeta serviceMeta; + private ServiceInstanceUpdate instanceMeta; + private EndpointMeta endpointMeta; + + @Override + public void build() { + if (nonNull(serviceMeta)) { + sourceReceiver.receive(serviceMeta); + } + if (nonNull(instanceMeta)) { + sourceReceiver.receive(instanceMeta); + } + if (nonNull(endpointMeta)) { + sourceReceiver.receive(endpointMeta); + } + } + + @Override + public LogSinkListener parse(final LogData.Builder logData, + final Message extraLog) { + Layer layer; + if (StringUtil.isNotEmpty(logData.getLayer())) { + layer = Layer.valueOf(logData.getLayer()); + } else { + layer = Layer.GENERAL; + } + final long timeBucket = TimeBucket.getTimeBucket(System.currentTimeMillis(), DownSampling.Minute); + // to service traffic + String serviceName = namingControl.formatServiceName(logData.getService()); + String serviceId = IDManager.ServiceID.buildId(serviceName, layer.isNormal()); + serviceMeta = new ServiceMeta(); + serviceMeta.setName(namingControl.formatServiceName(logData.getService())); + serviceMeta.setLayer(layer); + serviceMeta.setTimeBucket(timeBucket); + // to service instance traffic + if (StringUtil.isNotEmpty(logData.getServiceInstance())) { + instanceMeta = new ServiceInstanceUpdate(); + instanceMeta.setServiceId(serviceId); + instanceMeta.setName(namingControl.formatInstanceName(logData.getServiceInstance())); + instanceMeta.setTimeBucket(timeBucket); + + } + // to endpoint traffic + if (StringUtil.isNotEmpty(logData.getEndpoint())) { + endpointMeta = new EndpointMeta(); + endpointMeta.setServiceName(serviceName); + endpointMeta.setServiceNormal(true); + endpointMeta.setEndpoint(namingControl.formatEndpointName(serviceName, logData.getEndpoint())); + endpointMeta.setTimeBucket(timeBucket); + } + return this; + } + + public static class Factory implements LogSinkListenerFactory { + private final SourceReceiver sourceReceiver; + private final NamingControl namingControl; + + public Factory(ModuleManager moduleManager, LogAnalyzerModuleConfig moduleConfig) { + this.sourceReceiver = moduleManager.find(CoreModule.NAME) + .provider() + .getService(SourceReceiver.class); + this.namingControl = moduleManager.find(CoreModule.NAME) + .provider() + .getService(NamingControl.class); + } + + @Override + public LogSinkListener create() { + return new TrafficSinkListener(sourceReceiver, namingControl); + } + } +} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine b/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine new file mode 100644 index 000000000000..54d5a91d08b4 --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine @@ -0,0 +1,19 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule \ No newline at end of file diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider b/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider new file mode 100644 index 000000000000..8f00b261f68d --- /dev/null +++ b/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider @@ -0,0 +1,18 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleProvider \ No newline at end of file diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java b/test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java similarity index 100% rename from oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java rename to test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java b/test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java similarity index 100% rename from oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java rename to test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java diff --git a/oap-server/analyzer/log-analyzer/src/test/resources/log-mal-rules/placeholder.yaml b/test/script-compiler/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml similarity index 100% rename from oap-server/analyzer/log-analyzer/src/test/resources/log-mal-rules/placeholder.yaml rename to test/script-compiler/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml diff --git a/oap-server/analyzer/log-analyzer/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/test/script-compiler/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker similarity index 100% rename from oap-server/analyzer/log-analyzer/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker rename to test/script-compiler/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml b/test/script-compiler/mal-lal-v1-v2-checker/pom.xml similarity index 85% rename from oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml rename to test/script-compiler/mal-lal-v1-v2-checker/pom.xml index 523eb86d0fc9..d4e036f0a807 100644 --- a/oap-server/analyzer/mal-lal-v1-v2-checker/pom.xml +++ b/test/script-compiler/mal-lal-v1-v2-checker/pom.xml @@ -19,40 +19,41 @@ - analyzer + script-compiler org.apache.skywalking ${revision} 4.0.0 mal-lal-v1-v2-checker - Dual-path comparison tests: Groovy MAL/LAL (v1) vs transpiled Java MAL/LAL (v2) + Dual-path comparison tests: Groovy MAL/LAL (v1) vs compiler-generated Javassist MAL/LAL (v2) - + org.apache.skywalking - meter-analyzer + mal-v1-with-groovy ${project.version} test - + org.apache.skywalking - log-analyzer + lal-v1-with-groovy ${project.version} test - + org.apache.skywalking - mal-transpiler + meter-analyzer ${project.version} test + org.apache.skywalking - lal-transpiler + log-analyzer ${project.version} test diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java similarity index 100% rename from oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java rename to test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java similarity index 100% rename from oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java rename to test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java similarity index 100% rename from oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java rename to test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java diff --git a/oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java similarity index 100% rename from oap-server/analyzer/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java rename to test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java diff --git a/oap-server/analyzer/mal-transpiler/pom.xml b/test/script-compiler/mal-v1-with-groovy/pom.xml similarity index 87% rename from oap-server/analyzer/mal-transpiler/pom.xml rename to test/script-compiler/mal-v1-with-groovy/pom.xml index 4f45b67acb77..0fa3ad21065e 100644 --- a/oap-server/analyzer/mal-transpiler/pom.xml +++ b/test/script-compiler/mal-v1-with-groovy/pom.xml @@ -19,13 +19,13 @@ - analyzer + script-compiler org.apache.skywalking ${revision} 4.0.0 - mal-transpiler + mal-v1-with-groovy @@ -37,5 +37,9 @@ org.apache.groovy groovy + + io.vavr + vavr + diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java new file mode 100644 index 000000000000..a8b6a6662584 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java @@ -0,0 +1,383 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer; + +import com.google.common.base.Strings; +import com.google.common.collect.ImmutableMap; +import com.google.gson.JsonObject; +import io.vavr.Tuple; +import io.vavr.Tuple2; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.function.Predicate; +import java.util.stream.Stream; +import lombok.AccessLevel; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.StringUtils; +import org.apache.commons.text.CaseUtils; +import org.apache.skywalking.oap.meter.analyzer.dsl.DSL; +import org.apache.skywalking.oap.meter.analyzer.dsl.DownsamplingType; +import org.apache.skywalking.oap.meter.analyzer.dsl.Expression; +import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionParsingContext; +import org.apache.skywalking.oap.meter.analyzer.dsl.FilterExpression; +import org.apache.skywalking.oap.meter.analyzer.dsl.Result; +import org.apache.skywalking.oap.meter.analyzer.dsl.Sample; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.TimeBucket; +import org.apache.skywalking.oap.server.core.analysis.manual.endpoint.EndpointTraffic; +import org.apache.skywalking.oap.server.core.analysis.manual.instance.InstanceTraffic; +import org.apache.skywalking.oap.server.core.analysis.manual.relation.process.ProcessRelationClientSideMetrics; +import org.apache.skywalking.oap.server.core.analysis.manual.relation.process.ProcessRelationServerSideMetrics; +import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationClientSideMetrics; +import org.apache.skywalking.oap.server.core.analysis.manual.relation.service.ServiceRelationServerSideMetrics; +import org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTraffic; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; +import org.apache.skywalking.oap.server.core.analysis.meter.function.AcceptableValue; +import org.apache.skywalking.oap.server.core.analysis.meter.function.BucketedValues; +import org.apache.skywalking.oap.server.core.analysis.meter.function.PercentileArgument; +import org.apache.skywalking.oap.server.core.analysis.metrics.DataLabel; +import org.apache.skywalking.oap.server.core.analysis.metrics.DataTable; +import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor; + +import static com.google.common.collect.ImmutableMap.toImmutableMap; +import static java.util.Objects.requireNonNull; +import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.mapping; +import static java.util.stream.Collectors.toList; + +/** + * Analyzer analyses DSL expression with input samples, then to generate meter-system metrics. + */ +@Slf4j +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +@ToString(of = { + "metricName", + "expression" +}) +public class Analyzer { + + public static final Tuple2 NIL = Tuple.of("", null); + + public static Analyzer build(final String metricName, + final String filterExpression, + final String expression, + final MeterSystem meterSystem) { + Expression e = DSL.parse(metricName, expression); + FilterExpression filter = null; + if (!Strings.isNullOrEmpty(filterExpression)) { + filter = new FilterExpression(filterExpression); + } + ExpressionParsingContext ctx = e.parse(); + Analyzer analyzer = new Analyzer(metricName, filter, e, meterSystem, ctx); + analyzer.init(); + return analyzer; + } + + private List samples; + + private final String metricName; + + private final FilterExpression filterExpression; + + private final Expression expression; + + private final MeterSystem meterSystem; + + private final ExpressionParsingContext ctx; + + private MetricType metricType; + + private int[] percentiles; + + /** + * analyse intends to parse expression with input samples to meter-system metrics. + * + * @param sampleFamilies input samples. + */ + public void analyse(final ImmutableMap sampleFamilies) { + Map input = samples.stream() + .map(s -> Tuple.of(s, sampleFamilies.get(s))) + .filter(t -> t._2 != null) + .collect(toImmutableMap(t -> t._1, t -> t._2)); + if (input.size() < 1) { + if (log.isDebugEnabled()) { + log.debug("{} is ignored due to the lack of {}", expression, samples); + } + return; + } + if (filterExpression != null) { + input = filterExpression.filter(input); + if (input.isEmpty()) { + if (log.isDebugEnabled()) { + log.debug("{} is ignored due to mismatch of filter {}", expression, filterExpression); + } + return; + } + } + Result r = expression.run(input); + if (!r.isSuccess()) { + return; + } + SampleFamily.RunningContext ctx = r.getData().context; + Map meterSamples = ctx.getMeterSamples(); + meterSamples.forEach((meterEntity, ss) -> { + generateTraffic(meterEntity); + switch (metricType) { + case single: + AcceptableValue sv = meterSystem.buildMetrics(metricName, Long.class); + sv.accept(meterEntity, getValue(ss[0])); + send(sv, ss[0].getTimestamp()); + break; + case labeled: + AcceptableValue lv = meterSystem.buildMetrics(metricName, DataTable.class); + DataTable dt = new DataTable(); + // put all labels into the data table. + for (Sample each : ss) { + DataLabel dataLabel = new DataLabel(); + dataLabel.putAll(each.getLabels()); + dt.put(dataLabel, getValue(each)); + } + lv.accept(meterEntity, dt); + send(lv, ss[0].getTimestamp()); + break; + case histogram: + case histogramPercentile: + Stream.of(ss).map(s -> Tuple.of(getDataLabels(s.getLabels(), k -> !Objects.equals("le", k)), s)) + .collect(groupingBy(Tuple2::_1, mapping(Tuple2::_2, toList()))) + .forEach((dataLabel, subSs) -> { + if (subSs.size() < 1) { + return; + } + long[] bb = new long[subSs.size()]; + long[] vv = new long[bb.length]; + for (int i = 0; i < subSs.size(); i++) { + Sample s = subSs.get(i); + final double leVal = Double.parseDouble(s.getLabels().get("le")); + if (leVal == Double.NEGATIVE_INFINITY) { + bb[i] = Long.MIN_VALUE; + } else { + bb[i] = (long) leVal; + } + vv[i] = getValue(s); + } + BucketedValues bv = new BucketedValues(bb, vv); + bv.setLabels(dataLabel); + long time = subSs.get(0).getTimestamp(); + if (metricType == MetricType.histogram) { + AcceptableValue v = meterSystem.buildMetrics( + metricName, BucketedValues.class); + v.accept(meterEntity, bv); + send(v, time); + return; + } + AcceptableValue v = meterSystem.buildMetrics( + metricName, PercentileArgument.class); + v.accept(meterEntity, new PercentileArgument(bv, percentiles)); + send(v, time); + }); + break; + } + }); + } + + private long getValue(Sample sample) { + if (sample.getValue() <= 0.0) { + return 0L; + } + if (sample.getValue() < 1.0) { + return 1L; + } + return Math.round(sample.getValue()); + } + + private DataLabel getDataLabels(ImmutableMap labels, Predicate filter) { + DataLabel dataLabel = new DataLabel(); + labels.keySet().stream().filter(filter).forEach(k -> dataLabel.put(k, labels.get(k))); + return dataLabel; + } + + @RequiredArgsConstructor + private enum MetricType { + // metrics is aggregated by histogram function. + histogram("histogram"), + // metrics is aggregated by histogram based percentile function. + histogramPercentile("histogramPercentile"), + // metrics is aggregated by labeled function. + labeled("labeled"), + // metrics is aggregated by single value function. + single(""); + + private final String literal; + } + + private void init() { + this.samples = ctx.getSamples(); + if (ctx.isHistogram()) { + if (ctx.getPercentiles() != null && ctx.getPercentiles().length > 0) { + metricType = MetricType.histogramPercentile; + this.percentiles = ctx.getPercentiles(); + } else { + metricType = MetricType.histogram; + } + } else { + if (ctx.getLabels().isEmpty()) { + metricType = MetricType.single; + } else { + metricType = MetricType.labeled; + } + } + createMetric(ctx.getScopeType(), metricType.literal, ctx.getDownsampling()); + } + + private void createMetric(final ScopeType scopeType, + final String dataType, + final DownsamplingType downsamplingType) { + String downSamplingStr = CaseUtils.toCamelCase(downsamplingType.toString().toLowerCase(), false, '_'); + String functionName = String.format("%s%s", downSamplingStr, StringUtils.capitalize(dataType)); + meterSystem.create(metricName, functionName, scopeType); + } + + private void send(final AcceptableValue v, final long time) { + v.setTimeBucket(TimeBucket.getMinuteTimeBucket(time)); + meterSystem.doStreamingCalculation(v); + } + + private void generateTraffic(MeterEntity entity) { + if (entity.getDetectPoint() != null) { + switch (entity.getScopeType()) { + case SERVICE_RELATION: + serviceRelationTraffic(entity); + break; + case PROCESS_RELATION: + processRelationTraffic(entity); + break; + default: + } + } else { + toService(requireNonNull(entity.getServiceName()), entity.getLayer()); + } + + if (!com.google.common.base.Strings.isNullOrEmpty(entity.getInstanceName())) { + InstanceTraffic instanceTraffic = new InstanceTraffic(); + instanceTraffic.setName(entity.getInstanceName()); + instanceTraffic.setServiceId(entity.serviceId()); + instanceTraffic.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + instanceTraffic.setLastPingTimestamp(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + if (entity.getInstanceProperties() != null && !entity.getInstanceProperties().isEmpty()) { + final JsonObject properties = new JsonObject(); + entity.getInstanceProperties().forEach((k, v) -> properties.addProperty(k, v)); + instanceTraffic.setProperties(properties); + } + MetricsStreamProcessor.getInstance().in(instanceTraffic); + } + if (!com.google.common.base.Strings.isNullOrEmpty(entity.getEndpointName())) { + EndpointTraffic endpointTraffic = new EndpointTraffic(); + endpointTraffic.setName(entity.getEndpointName()); + endpointTraffic.setServiceId(entity.serviceId()); + endpointTraffic.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + endpointTraffic.setLastPingTimestamp(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + MetricsStreamProcessor.getInstance().in(endpointTraffic); + } + } + + private void toService(String serviceName, Layer layer) { + ServiceTraffic s = new ServiceTraffic(); + s.setName(requireNonNull(serviceName)); + s.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + s.setLayer(layer); + MetricsStreamProcessor.getInstance().in(s); + } + + private void serviceRelationTraffic(MeterEntity entity) { + switch (entity.getDetectPoint()) { + case SERVER: + entity.setServiceName(entity.getDestServiceName()); + toService(requireNonNull(entity.getDestServiceName()), entity.getLayer()); + serviceRelationServerSide(entity); + break; + case CLIENT: + entity.setServiceName(entity.getSourceServiceName()); + toService(requireNonNull(entity.getSourceServiceName()), entity.getLayer()); + serviceRelationClientSide(entity); + break; + default: + } + } + + private void serviceRelationServerSide(MeterEntity entity) { + ServiceRelationServerSideMetrics metrics = new ServiceRelationServerSideMetrics(); + metrics.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + metrics.setSourceServiceId(entity.sourceServiceId()); + metrics.setDestServiceId(entity.destServiceId()); + metrics.getComponentIds().add(entity.getComponentId()); + metrics.setEntityId(entity.id()); + MetricsStreamProcessor.getInstance().in(metrics); + } + + private void serviceRelationClientSide(MeterEntity entity) { + ServiceRelationClientSideMetrics metrics = new ServiceRelationClientSideMetrics(); + metrics.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + metrics.setSourceServiceId(entity.sourceServiceId()); + metrics.setDestServiceId(entity.destServiceId()); + metrics.getComponentIds().add(entity.getComponentId()); + metrics.setEntityId(entity.id()); + MetricsStreamProcessor.getInstance().in(metrics); + } + + private void processRelationTraffic(MeterEntity entity) { + switch (entity.getDetectPoint()) { + case SERVER: + processRelationServerSide(entity); + break; + case CLIENT: + processRelationClientSide(entity); + break; + default: + } + } + + private void processRelationServerSide(MeterEntity entity) { + ProcessRelationServerSideMetrics metrics = new ProcessRelationServerSideMetrics(); + metrics.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + metrics.setServiceInstanceId(entity.serviceInstanceId()); + metrics.setSourceProcessId(entity.getSourceProcessId()); + metrics.setDestProcessId(entity.getDestProcessId()); + metrics.setEntityId(entity.id()); + metrics.setComponentId(entity.getComponentId()); + MetricsStreamProcessor.getInstance().in(metrics); + } + + private void processRelationClientSide(MeterEntity entity) { + ProcessRelationClientSideMetrics metrics = new ProcessRelationClientSideMetrics(); + metrics.setTimeBucket(TimeBucket.getMinuteTimeBucket(System.currentTimeMillis())); + metrics.setServiceInstanceId(entity.serviceInstanceId()); + metrics.setSourceProcessId(entity.getSourceProcessId()); + metrics.setDestProcessId(entity.getDestProcessId()); + metrics.setEntityId(entity.id()); + metrics.setComponentId(entity.getComponentId()); + MetricsStreamProcessor.getInstance().in(metrics); + } + +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java new file mode 100644 index 000000000000..5c89f42b5edf --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer; + +import com.google.common.base.Preconditions; +import com.google.common.base.Strings; +import com.google.common.collect.ImmutableMap; +import io.vavr.control.Try; +import java.util.List; +import java.util.StringJoiner; +import java.util.stream.Stream; +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.StringUtils; +import org.apache.skywalking.oap.meter.analyzer.dsl.DSL; +import org.apache.skywalking.oap.meter.analyzer.dsl.Expression; +import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionParsingException; +import org.apache.skywalking.oap.meter.analyzer.dsl.Result; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem; + +import static java.util.stream.Collectors.toList; + +/** + * MetricConvert converts {@link SampleFamily} collection to meter-system metrics, then store them to backend storage. + */ +@Slf4j +public class MetricConvert { + + public static Stream log(Try t, String debugMessage) { + return t + .onSuccess(i -> log.debug(debugMessage + " :{}", i)) + .onFailure(e -> log.debug(debugMessage + " failed", e)) + .toJavaStream(); + } + + private final List analyzers; + + public MetricConvert(MetricRuleConfig rule, MeterSystem service) { + Preconditions.checkState(!Strings.isNullOrEmpty(rule.getMetricPrefix())); + // init expression script + if (StringUtils.isNotEmpty(rule.getInitExp())) { + handleInitExp(rule.getInitExp()); + } + this.analyzers = rule.getMetricsRules().stream().map( + r -> buildAnalyzer( + formatMetricName(rule, r.getName()), + rule.getFilter(), + formatExp(rule.getExpPrefix(), rule.getExpSuffix(), r.getExp()), + service + ) + ).collect(toList()); + } + + Analyzer buildAnalyzer(final String metricsName, + final String filter, + final String exp, + final MeterSystem service) { + return Analyzer.build( + metricsName, + filter, + exp, + service + ); + } + + private String formatExp(final String expPrefix, String expSuffix, String exp) { + String ret = exp; + if (!Strings.isNullOrEmpty(expPrefix)) { + ret = String.format("(%s.%s)", StringUtils.substringBefore(exp, "."), expPrefix); + final String after = StringUtils.substringAfter(exp, "."); + if (!Strings.isNullOrEmpty(after)) { + ret = String.format("(%s.%s)", ret, after); + } + } + if (!Strings.isNullOrEmpty(expSuffix)) { + ret = String.format("(%s).%s", ret, expSuffix); + } + return ret; + } + + /** + * toMeter transforms {@link SampleFamily} collection to meter-system metrics. + * + * @param sampleFamilies {@link SampleFamily} collection. + */ + public void toMeter(final ImmutableMap sampleFamilies) { + Preconditions.checkNotNull(sampleFamilies); + if (sampleFamilies.size() < 1) { + return; + } + for (Analyzer each : analyzers) { + try { + each.analyse(sampleFamilies); + } catch (Throwable t) { + log.error("Analyze {} error", each, t); + } + } + } + + private String formatMetricName(MetricRuleConfig rule, String meterRuleName) { + StringJoiner metricName = new StringJoiner("_"); + metricName.add(rule.getMetricPrefix()).add(meterRuleName); + return metricName.toString(); + } + + private void handleInitExp(String exp) { + Expression e = DSL.parse(null, exp); + final Result result = e.run(ImmutableMap.of()); + if (!result.isSuccess() && result.isThrowable()) { + throw new ExpressionParsingException( + "failed to execute init expression: " + exp + ", error:" + result.getError()); + } + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java new file mode 100644 index 000000000000..2e75b1d89cda --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java @@ -0,0 +1,66 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer; + +import java.util.List; + +/** + * Metrics rules convert to meter system. + */ +public interface MetricRuleConfig { + + /** + * Get metrics name prefix + */ + String getMetricPrefix(); + + /** + * Get MAL expression suffix + */ + String getExpSuffix(); + + /** + * Get MAL expression prefix + */ + String getExpPrefix(); + + /** + * Get all rules + */ + List getMetricsRules(); + + String getFilter(); + + /** + * Get the init expression script + */ + String getInitExp(); + + interface RuleConfig { + /** + * Get definition metrics name + */ + String getName(); + + /** + * Build metrics MAL + */ + String getExp(); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java new file mode 100644 index 000000000000..e723b6add348 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import com.google.common.collect.ImmutableList; +import groovy.lang.Binding; +import groovy.lang.GString; +import groovy.lang.GroovyShell; +import groovy.util.DelegatingScript; +import java.lang.reflect.Array; +import java.util.List; +import java.util.Map; + +import org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry; +import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.source.DetectPoint; +import org.codehaus.groovy.ast.stmt.DoWhileStatement; +import org.codehaus.groovy.ast.stmt.ForStatement; +import org.codehaus.groovy.ast.stmt.Statement; +import org.codehaus.groovy.ast.stmt.WhileStatement; +import org.codehaus.groovy.control.CompilerConfiguration; +import org.codehaus.groovy.control.customizers.ImportCustomizer; +import org.codehaus.groovy.control.customizers.SecureASTCustomizer; + +/** + * DSL combines methods to parse groovy based DSL expression. + */ +public final class DSL { + + /** + * Parse string literal to Expression object, which can be reused. + * + * @param metricName the name of metric defined in mal rule + * @param expression string literal represents the DSL expression. + * @return Expression object could be executed. + */ + public static Expression parse(final String metricName, final String expression) { + CompilerConfiguration cc = new CompilerConfiguration(); + cc.setScriptBaseClass(DelegatingScript.class.getName()); + ImportCustomizer icz = new ImportCustomizer(); + icz.addImport("K8sRetagType", K8sRetagType.class.getName()); + icz.addImport("DetectPoint", DetectPoint.class.getName()); + icz.addImport("Layer", Layer.class.getName()); + icz.addImport("ProcessRegistry", ProcessRegistry.class.getName()); + cc.addCompilationCustomizers(icz); + + final SecureASTCustomizer secureASTCustomizer = new SecureASTCustomizer(); + secureASTCustomizer.setDisallowedStatements( + ImmutableList.>builder() + .add(WhileStatement.class) + .add(DoWhileStatement.class) + .add(ForStatement.class) + .build()); + // noinspection rawtypes + secureASTCustomizer.setAllowedReceiversClasses( + ImmutableList.builder() + .add(Object.class) + .add(Map.class) + .add(List.class) + .add(Array.class) + .add(K8sRetagType.class) + .add(DetectPoint.class) + .add(Layer.class) + .add(ProcessRegistry.class) + .add(GString.class) + .add(String.class) + .build()); + cc.addCompilationCustomizers(secureASTCustomizer); + + GroovyShell sh = new GroovyShell(new Binding(), cc); + DelegatingScript script = (DelegatingScript) sh.parse(expression); + return new Expression(metricName, expression, script); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java new file mode 100644 index 000000000000..c8f0006fbb52 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +/** + * DownsamplingType indicates the downsampling type of meter function + */ +public enum DownsamplingType { + AVG, SUM, LATEST, SUM_PER_MIN, MAX, MIN +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java new file mode 100644 index 000000000000..1eac8973ca46 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java @@ -0,0 +1,44 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +@Getter +@RequiredArgsConstructor +@ToString +public class EndpointEntityDescription implements EntityDescription { + private final ScopeType scopeType = ScopeType.ENDPOINT; + private final List serviceKeys; + private final List endpointKeys; + private final Layer layer; + private final String delimiter; + + @Override + public List getLabelKeys() { + return Stream.concat(this.serviceKeys.stream(), this.endpointKeys.stream()).collect(Collectors.toList()); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java new file mode 100644 index 000000000000..24e09090fb98 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import java.util.List; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +public interface EntityDescription { + ScopeType getScopeType(); + + List getLabelKeys(); +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java new file mode 100644 index 000000000000..83f1e5f87f23 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import java.util.List; +import java.util.Map; +import java.util.function.Function; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +@Getter +@RequiredArgsConstructor +@ToString +public class InstanceEntityDescription implements EntityDescription { + private final ScopeType scopeType = ScopeType.SERVICE_INSTANCE; + private final List serviceKeys; + private final List instanceKeys; + private final Layer layer; + private final String serviceDelimiter; + private final String instanceDelimiter; + private final Function, Map> propertiesExtractor; + + @Override + public List getLabelKeys() { + return Stream.concat(this.serviceKeys.stream(), this.instanceKeys.stream()).collect(Collectors.toList()); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java new file mode 100644 index 000000000000..9759061eabcc --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import com.google.common.collect.ImmutableList; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +import java.util.List; + +@Getter +@RequiredArgsConstructor +@ToString +public class ProcessEntityDescription implements EntityDescription { + private final ScopeType scopeType = ScopeType.PROCESS; + private final List serviceKeys; + private final List serviceInstanceKeys; + private final List processKeys; + private final String layerKey; + private final String delimiter; + + @Override + public List getLabelKeys() { + return ImmutableList.builder() + .addAll(serviceKeys) + .addAll(serviceInstanceKeys) + .addAll(processKeys) + .add(layerKey) + .build(); + } +} \ No newline at end of file diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java new file mode 100644 index 000000000000..1e3533e30694 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import com.google.common.collect.ImmutableList; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +import java.util.List; + +@Getter +@RequiredArgsConstructor +@ToString +public class ProcessRelationEntityDescription implements EntityDescription { + private final ScopeType scopeType = ScopeType.PROCESS_RELATION; + private final List serviceKeys; + private final List instanceKeys; + private final String sourceProcessIdKey; + private final String destProcessIdKey; + private final String detectPointKey; + private final String componentKey; + private final String delimiter; + + @Override + public List getLabelKeys() { + return ImmutableList.builder() + .addAll(serviceKeys) + .addAll(instanceKeys) + .add(detectPointKey, sourceProcessIdKey, destProcessIdKey, componentKey).build(); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java new file mode 100644 index 000000000000..b1fc80c30826 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import java.util.List; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; + +@Getter +@RequiredArgsConstructor +@ToString +public class ServiceEntityDescription implements EntityDescription { + private final ScopeType scopeType = ScopeType.SERVICE; + private final List serviceKeys; + private final Layer layer; + private final String delimiter; + + @Override + public List getLabelKeys() { + return serviceKeys; + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java new file mode 100644 index 000000000000..1adf7fd9aab7 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java @@ -0,0 +1,53 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription; + +import java.util.List; +import com.google.common.collect.ImmutableList; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; +import org.apache.skywalking.oap.server.core.source.DetectPoint; +import org.apache.skywalking.oap.server.library.util.StringUtil; + +@Getter +@RequiredArgsConstructor +@ToString +public class ServiceRelationEntityDescription implements EntityDescription { + private final ScopeType scopeType = ScopeType.SERVICE_RELATION; + private final List sourceServiceKeys; + private final List destServiceKeys; + private final DetectPoint detectPoint; + private final Layer layer; + private final String delimiter; + private final String componentIdKey; + + @Override + public List getLabelKeys() { + final ImmutableList.Builder builder = ImmutableList.builder() + .addAll(this.sourceServiceKeys) + .addAll(this.destServiceKeys); + if (StringUtil.isNotEmpty(componentIdKey)) { + builder.add(componentIdKey); + } + return builder.build(); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java new file mode 100644 index 000000000000..02912adb05f0 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import com.google.common.collect.ImmutableMap; +import groovy.lang.ExpandoMetaClass; +import groovy.lang.GroovyObjectSupport; +import groovy.util.DelegatingScript; +import java.time.Instant; +import java.util.Map; +import lombok.RequiredArgsConstructor; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; + +/** + * Expression is a reusable monadic container type which represents a DSL expression. + */ +@Slf4j +@ToString(of = {"literal"}) +public class Expression { + private static final ThreadLocal> PROPERTY_REPOSITORY = new ThreadLocal<>(); + + private final String metricName; + + private final String literal; + + private final DelegatingScript expression; + + public Expression(final String metricName, final String literal, final DelegatingScript expression) { + this.metricName = metricName; + this.literal = literal; + this.expression = expression; + this.empower(); + } + + /** + * Parse the expression statically. + * + * @return Parsed context of the expression. + */ + public ExpressionParsingContext parse() { + try (ExpressionParsingContext ctx = ExpressionParsingContext.create()) { + Result r = run(ImmutableMap.of()); + if (!r.isSuccess() && r.isThrowable()) { + throw new ExpressionParsingException( + "failed to parse expression: " + literal + ", error:" + r.getError()); + } + if (log.isDebugEnabled()) { + log.debug("\"{}\" is parsed", literal); + } + ctx.validate(literal); + return ctx; + } + } + + /** + * Run the expression with a data map. + * + * @param sampleFamilies a data map includes all of candidates to be analysis. + * @return The result of execution. + */ + public Result run(final Map sampleFamilies) { + PROPERTY_REPOSITORY.set(sampleFamilies); + try { + SampleFamily sf = (SampleFamily) expression.run(); + if (sf == SampleFamily.EMPTY) { + if (!ExpressionParsingContext.get().isPresent()) { + if (log.isDebugEnabled()) { + log.debug("result of {} is empty by \"{}\"", sampleFamilies, literal); + } + } + return Result.fail("Parsed result is an EMPTY sample family"); + } + return Result.success(sf); + } catch (Throwable t) { + log.error("failed to run \"{}\"", literal, t); + return Result.fail(t); + } finally { + PROPERTY_REPOSITORY.remove(); + } + } + + private void empower() { + expression.setDelegate(new ExpressionDelegate(metricName, literal)); + extendNumber(Number.class); + } + + private void extendNumber(Class clazz) { + ExpandoMetaClass expando = new ExpandoMetaClass(clazz, true, false); + expando.registerInstanceMethod("plus", new NumberClosure(this, (n, s) -> s.plus(n))); + expando.registerInstanceMethod("minus", new NumberClosure(this, (n, s) -> s.minus(n).negative())); + expando.registerInstanceMethod("multiply", new NumberClosure(this, (n, s) -> s.multiply(n))); + expando.registerInstanceMethod("div", new NumberClosure(this, (n, s) -> s.newValue(v -> n.doubleValue() / v))); + expando.initialize(); + } + + @RequiredArgsConstructor + @SuppressWarnings("unused") // used in MAL expressions + private static class ExpressionDelegate extends GroovyObjectSupport { + public static final DownsamplingType AVG = DownsamplingType.AVG; + public static final DownsamplingType SUM = DownsamplingType.SUM; + public static final DownsamplingType LATEST = DownsamplingType.LATEST; + public static final DownsamplingType SUM_PER_MIN = DownsamplingType.SUM_PER_MIN; + public static final DownsamplingType MAX = DownsamplingType.MAX; + public static final DownsamplingType MIN = DownsamplingType.MIN; + + private final String metricName; + private final String literal; + + public SampleFamily propertyMissing(String sampleName) { + ExpressionParsingContext.get().ifPresent(ctx -> { + if (!ctx.samples.contains(sampleName)) { + ctx.samples.add(sampleName); + } + }); + Map sampleFamilies = PROPERTY_REPOSITORY.get(); + if (sampleFamilies == null) { + return SampleFamily.EMPTY; + } + if (sampleFamilies.containsKey(sampleName)) { + SampleFamily sampleFamily = sampleFamilies.get(sampleName); + sampleFamily.context.setMetricName(this.metricName); + return sampleFamily; + } + if (ExpressionParsingContext.get().isEmpty()) { + log.warn("{} referred by \"{}\" doesn't exist in {}", sampleName, literal, sampleFamilies.keySet()); + } + return SampleFamily.EMPTY; + } + + public Number time() { + return Instant.now().getEpochSecond(); + } + + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java rename to test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java new file mode 100644 index 000000000000..a3ab8a37d9ed --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java @@ -0,0 +1,28 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +/** + * ExpressionParsingException is throw in expression parsing phase. + */ +public class ExpressionParsingException extends RuntimeException { + public ExpressionParsingException(final String message) { + super(message); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java new file mode 100644 index 000000000000..9b532bf72343 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java @@ -0,0 +1,58 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import groovy.lang.Closure; +import groovy.lang.GroovyShell; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; + +@Slf4j +@ToString(of = {"literal"}) +public class FilterExpression { + private final String literal; + private final Closure filterClosure; + + @SuppressWarnings("unchecked") + public FilterExpression(final String literal) { + this.literal = literal; + + GroovyShell sh = new GroovyShell(); + filterClosure = (Closure) sh.evaluate(literal); + } + + public Map filter(final Map sampleFamilies) { + try { + Map result = new HashMap<>(); + for (Map.Entry entry : sampleFamilies.entrySet()) { + SampleFamily afterFilter = entry.getValue().filter(filterClosure); + if (!Objects.equals(afterFilter, SampleFamily.EMPTY)) { + result.put(entry.getKey(), afterFilter); + } + } + return result; + } catch (Throwable t) { + log.error("failed to run \"{}\"", literal, t); + } + return sampleFamilies; + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java new file mode 100644 index 000000000000..cfaf1d5beee5 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.Map; + +/** + * Pure Java replacement for Groovy-based MAL DelegatingScript. + * Each transpiled MAL expression implements this interface. + */ +@FunctionalInterface +public interface MalExpression { + SampleFamily run(Map samples); +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java new file mode 100644 index 000000000000..9d8eaa259e92 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java @@ -0,0 +1,30 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.Map; + +/** + * Pure Java replacement for Groovy Closure-based MAL filter expressions. + * Each transpiled filter expression implements this interface. + */ +@FunctionalInterface +public interface MalFilter { + boolean test(Map tags); +} diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java rename to test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java new file mode 100644 index 000000000000..195219653b26 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import lombok.AccessLevel; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.ToString; + +/** + * Result indicates the parsing result of expression. + */ +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +@EqualsAndHashCode +@ToString +@Getter +public class Result { + + /** + * fail is a static factory method builds failed result based on {@link Throwable}. + * + * @param throwable to build failed result. + * @return failed result. + */ + public static Result fail(final Throwable throwable) { + return new Result(false, true, throwable.getMessage(), SampleFamily.EMPTY); + } + + /** + * fail is a static factory method builds failed result based on error message. + * + * @param message is the error details why the result is failed. + * @return failed result. + */ + public static Result fail(String message) { + return new Result(false, false, message, SampleFamily.EMPTY); + } + + /** + * fail is a static factory method builds failed result. + * + * @return failed result. + */ + public static Result fail() { + return new Result(false, false, null, SampleFamily.EMPTY); + } + + /** + * success is a static factory method builds successful result. + * + * @param sf is the parsed result. + * @return successful result. + */ + public static Result success(SampleFamily sf) { + return new Result(true, false, null, sf); + } + + private final boolean success; + + private final boolean isThrowable; + + private final String error; + + private final SampleFamily data; +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java new file mode 100644 index 000000000000..9a6760fde6b7 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java @@ -0,0 +1,60 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import com.google.common.collect.ImmutableMap; +import io.vavr.Function2; +import io.vavr.Tuple2; +import java.time.Duration; +import java.util.function.Function; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.ToString; +import org.apache.skywalking.oap.meter.analyzer.dsl.counter.CounterWindow; + +/** + * Sample represents the metric data point in a range of time. + */ +@Builder(toBuilder = true) +@EqualsAndHashCode +@ToString +@Getter +public class Sample { + final String name; + final ImmutableMap labels; + final double value; + final long timestamp; + + Sample newValue(Function transform) { + return toBuilder().value(transform.apply(value)).build(); + } + + Sample increase(String range, String metricName, Function2 transform) { + Tuple2 i = CounterWindow.INSTANCE.increase(metricName, labels, value, Duration.parse(range).toMillis(), timestamp); + double nv = transform.apply(i._2, i._1); + return newValue(ignored -> nv); + } + + Sample increase(String metricName, Function2 transform) { + Tuple2 i = CounterWindow.INSTANCE.pop(metricName, labels, value, timestamp); + double nv = transform.apply(i._2, i._1); + return newValue(ignored -> nv); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java new file mode 100644 index 000000000000..5a979d0a381d --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java @@ -0,0 +1,985 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import static java.util.function.UnaryOperator.identity; +import static java.util.stream.Collectors.groupingBy; +import static java.util.stream.Collectors.mapping; +import static java.util.stream.Collectors.toList; +import static com.google.common.collect.ImmutableMap.toImmutableMap; + +import org.apache.commons.lang3.StringUtils; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.EndpointEntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.EntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.InstanceEntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.ProcessEntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.ProcessRelationEntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.ServiceEntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.EntityDescription.ServiceRelationEntityDescription; +import org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt.K8sRetagType; +import org.apache.skywalking.oap.server.core.Const; +import org.apache.skywalking.oap.server.core.UnexpectedException; +import org.apache.skywalking.oap.server.core.analysis.Layer; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity; +import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; +import org.apache.skywalking.oap.server.core.source.DetectPoint; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.function.DoubleBinaryOperator; +import java.util.function.Function; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import com.google.common.base.Preconditions; +import com.google.common.base.Strings; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.Maps; +import groovy.lang.Closure; +import io.vavr.Function2; +import io.vavr.Function3; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.DecorateFunction; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.ForEachFunction; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.PropertiesExtractor; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.SampleFilter; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyFunctions.TagFunction; +import lombok.AccessLevel; +import lombok.Builder; +import lombok.EqualsAndHashCode; +import lombok.Getter; +import lombok.RequiredArgsConstructor; +import lombok.Setter; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.server.library.util.StringUtil; + +/** + * SampleFamily represents a collection of {@link Sample}. + */ +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +@EqualsAndHashCode +@ToString +@Slf4j +public class SampleFamily { + public static final SampleFamily EMPTY = new SampleFamily(new Sample[0], RunningContext.EMPTY); + + static SampleFamily build(RunningContext ctx, Sample... samples) { + Preconditions.checkNotNull(samples); + Preconditions.checkArgument(samples.length > 0); + samples = Arrays.stream(samples).filter(sample -> !Double.isNaN(sample.getValue())).toArray(Sample[]::new); + if (samples.length == 0) { + return EMPTY; + } + return new SampleFamily(samples, Optional.ofNullable(ctx).orElseGet(RunningContext::instance)); + } + + public final Sample[] samples; + + public final RunningContext context; + + /** + * Following operations are used in DSL + */ + + /* tag filter operations*/ + public SampleFamily tagEqual(String... labels) { + return match(labels, InternalOps::stringComp); + } + + public SampleFamily tagNotEqual(String[] labels) { + return match(labels, (sv, lv) -> !InternalOps.stringComp(sv, lv)); + } + + public SampleFamily tagMatch(String[] labels) { + return match(labels, String::matches); + } + + public SampleFamily tagNotMatch(String[] labels) { + return match(labels, (sv, lv) -> !sv.matches(lv)); + } + + /* value filter operations*/ + public SampleFamily valueEqual(double compValue) { + return valueMatch(CompType.EQUAL, compValue, InternalOps::doubleComp); + } + + public SampleFamily valueNotEqual(double compValue) { + return valueMatch(CompType.NOT_EQUAL, compValue, InternalOps::doubleComp); + } + + public SampleFamily valueGreater(double compValue) { + return valueMatch(CompType.GREATER, compValue, InternalOps::doubleComp); + } + + public SampleFamily valueGreaterEqual(double compValue) { + return valueMatch(CompType.GREATER_EQUAL, compValue, InternalOps::doubleComp); + } + + public SampleFamily valueLess(double compValue) { + return valueMatch(CompType.LESS, compValue, InternalOps::doubleComp); + } + + public SampleFamily valueLessEqual(double compValue) { + return valueMatch(CompType.LESS_EQUAL, compValue, InternalOps::doubleComp); + } + + /* Binary operator overloading*/ + public SampleFamily plus(Number number) { + return newValue(v -> v + number.doubleValue()); + } + + public SampleFamily minus(Number number) { + return newValue(v -> v - number.doubleValue()); + } + + public SampleFamily multiply(Number number) { + return newValue(v -> v * number.doubleValue()); + } + + public SampleFamily div(Number number) { + return newValue(v -> v / number.doubleValue()); + } + + public SampleFamily negative() { + return newValue(v -> -v); + } + + public SampleFamily plus(SampleFamily another) { + if (this == EMPTY && another == EMPTY) { + return SampleFamily.EMPTY; + } + if (this == EMPTY) { + return another; + } + if (another == EMPTY) { + return this; + } + return newValue(another, Double::sum); + } + + public SampleFamily minus(SampleFamily another) { + if (this == EMPTY && another == EMPTY) { + return SampleFamily.EMPTY; + } + if (this == EMPTY) { + return another.negative(); + } + if (another == EMPTY) { + return this; + } + return newValue(another, (a, b) -> a - b); + } + + public SampleFamily multiply(SampleFamily another) { + if (this == EMPTY || another == EMPTY) { + return SampleFamily.EMPTY; + } + return newValue(another, (a, b) -> a * b); + } + + public SampleFamily div(SampleFamily another) { + if (this == EMPTY) { + return SampleFamily.EMPTY; + } + if (another == EMPTY) { + return div(0.0); + } + return newValue(another, (a, b) -> a / b); + } + + /* Aggregation operators */ + public SampleFamily sum(List by) { + return aggregate(by, Double::sum); + } + + public SampleFamily max(List by) { + return aggregate(by, Double::max); + } + + public SampleFamily min(List by) { + return aggregate(by, Double::min); + } + + public SampleFamily avg(List by) { + ExpressionParsingContext.get().ifPresent(ctx -> ctx.aggregationLabels.addAll(by)); + if (this == EMPTY) { + return EMPTY; + } + if (by == null) { + double result = Arrays.stream(samples).mapToDouble(Sample::getValue).average().orElse(0.0D); + return SampleFamily.build( + this.context, InternalOps.newSample(samples[0].name, ImmutableMap.of(), samples[0].timestamp, result)); + } + + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .collect(groupingBy(it -> InternalOps.getLabels(by, it), mapping(identity(), toList()))) + .entrySet().stream() + .map(entry -> InternalOps.newSample( + entry.getValue().get(0).getName(), + entry.getKey(), + entry.getValue().get(0).getTimestamp(), + entry.getValue().stream().mapToDouble(Sample::getValue).average().orElse(0.0D) + )) + .toArray(Sample[]::new) + ); + } + + public SampleFamily count(List by) { + ExpressionParsingContext.get().ifPresent(ctx -> ctx.aggregationLabels.addAll(by)); + if (this == EMPTY) { + return EMPTY; + } + if (by == null) { + long result = Arrays.stream(samples).count(); + return SampleFamily.build( + this.context, InternalOps.newSample(samples[0].name, ImmutableMap.of(), samples[0].timestamp, result)); + } + + if (by.size() == 1) { + Set set = Arrays + .stream(samples) + .map(sample -> sample.labels.get(by.get(0))) + .filter(StringUtils::isNotBlank) + .collect(Collectors.toSet()); + + return SampleFamily.build( + this.context, InternalOps.newSample(samples[0].name, ImmutableMap.of(), samples[0].timestamp, set.size())); + } + + Stream, List>> stream = Arrays + .stream(samples) + .filter(sample -> sample.labels.keySet().containsAll(by)) + .collect(groupingBy(it -> InternalOps.getLabels(by, it))) + .entrySet() + .stream() + .map(entry -> InternalOps.newSample( + entry.getValue().get(0).getName(), + entry.getKey(), + entry.getValue().get(0).getTimestamp(), + entry.getValue().size())) + .collect(groupingBy(it -> InternalOps.groupByExcludedLabel(by.get(by.size() - 1), it), mapping(identity(), toList()))) + .entrySet() + .stream(); + + Sample[] array = stream + .map(entry -> InternalOps.newSample( + entry.getValue().get(0).getName(), + entry.getKey(), + entry.getValue().get(0).getTimestamp(), + entry.getValue().size() + )) + .toArray(Sample[]::new); + + SampleFamily sampleFamily = SampleFamily.build( + this.context, + array + ); + return sampleFamily; + } + + protected SampleFamily aggregate(List by, DoubleBinaryOperator aggregator) { + ExpressionParsingContext.get().ifPresent(ctx -> ctx.aggregationLabels.addAll(by)); + if (this == EMPTY) { + return EMPTY; + } + if (by == null) { + double result = Arrays.stream(samples).mapToDouble(s -> s.value).reduce(aggregator).orElse(0.0D); + return SampleFamily.build( + this.context, InternalOps.newSample(samples[0].name, ImmutableMap.of(), samples[0].timestamp, result)); + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .collect(groupingBy(it -> InternalOps.getLabels(by, it), mapping(identity(), toList()))) + .entrySet().stream() + .map(entry -> InternalOps.newSample( + entry.getValue().get(0).getName(), + entry.getKey(), + entry.getValue().get(0).getTimestamp(), + entry.getValue().stream().mapToDouble(Sample::getValue).reduce(aggregator).orElse(0.0D) + )) + .toArray(Sample[]::new) + ); + } + + /* Function */ + public SampleFamily increase(String range) { + Preconditions.checkArgument(!Strings.isNullOrEmpty(range)); + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .map(sample -> sample.increase( + range, + context.metricName, + (lowerBoundValue, unused) -> sample.value - lowerBoundValue + )) + .toArray(Sample[]::new) + ); + } + + public SampleFamily rate(String range) { + Preconditions.checkArgument(!Strings.isNullOrEmpty(range)); + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .map(sample -> sample.increase( + range, + context.metricName, + (lowerBoundValue, lowerBoundTime) -> { + final long timeDiff = (sample.timestamp - lowerBoundTime) / 1000; + return timeDiff < 1L ? 0.0 : (sample.value - lowerBoundValue) / timeDiff; + } + )) + .toArray(Sample[]::new) + ); + } + + public SampleFamily irate() { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .map(sample -> sample.increase( + context.metricName, + (lowerBoundValue, lowerBoundTime) -> { + final long timeDiff = (sample.timestamp - lowerBoundTime) / 1000; + return timeDiff < 1L ? 0.0 : (sample.value - lowerBoundValue) / timeDiff; + } + )) + .toArray(Sample[]::new) + ); + } + + @SuppressWarnings(value = "unchecked") + public SampleFamily tag(Closure cl) { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .map(sample -> { + Object delegate = new Object(); + Closure c = cl.rehydrate(delegate, sample, delegate); + Map arg = Maps.newHashMap(sample.labels); + Object r = c.call(arg); + return sample.toBuilder() + .labels( + ImmutableMap.copyOf( + Optional.ofNullable((r instanceof Map) ? (Map) r : null) + .orElse(arg))) + .build(); + }).toArray(Sample[]::new) + ); + } + + @SuppressWarnings(value = "unchecked") + public SampleFamily tag(TagFunction fn) { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Arrays.stream(samples) + .map(sample -> { + Map arg = Maps.newHashMap(sample.labels); + Map r = fn.apply(arg); + return sample.toBuilder() + .labels( + ImmutableMap.copyOf( + Optional.ofNullable(r).orElse(arg))) + .build(); + }).toArray(Sample[]::new) + ); + } + + public SampleFamily filter(Closure filter) { + if (this == EMPTY) { + return EMPTY; + } + final Sample[] filtered = Arrays.stream(samples) + .filter(it -> filter.call(it.labels)) + .toArray(Sample[]::new); + if (filtered.length == 0) { + return EMPTY; + } + return SampleFamily.build(context, filtered); + } + + public SampleFamily filter(SampleFilter filter) { + if (this == EMPTY) { + return EMPTY; + } + final Sample[] filtered = Arrays.stream(samples) + .filter(it -> filter.test(it.labels)) + .toArray(Sample[]::new); + if (filtered.length == 0) { + return EMPTY; + } + return SampleFamily.build(context, filtered); + } + + /* k8s retags*/ + public SampleFamily retagByK8sMeta(String newLabelName, + K8sRetagType type, + String existingLabelName, + String namespaceLabelName) { + Preconditions.checkArgument(!Strings.isNullOrEmpty(newLabelName)); + Preconditions.checkArgument(!Strings.isNullOrEmpty(existingLabelName)); + Preconditions.checkArgument(!Strings.isNullOrEmpty(namespaceLabelName)); + if (this == EMPTY) { + return EMPTY; + } + + return SampleFamily.build( + this.context, type.execute(samples, newLabelName, existingLabelName, namespaceLabelName)); + } + + public SampleFamily histogram() { + return histogram("le", this.context.defaultHistogramBucketUnit); + } + + public SampleFamily histogram(String le) { + return histogram(le, this.context.defaultHistogramBucketUnit); + } + + public SampleFamily histogram(String le, TimeUnit unit) { + long scale = unit.toMillis(1); + Preconditions.checkArgument(scale > 0); + ExpressionParsingContext.get().ifPresent(ctx -> ctx.isHistogram = true); + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build( + this.context, + Stream.concat( + Arrays.stream(samples).filter(s -> !s.labels.containsKey(le)), + Arrays.stream(samples) + .filter(s -> s.labels.containsKey(le)) + .sorted(Comparator.comparingDouble(s -> Double.parseDouble(s.labels.get(le)))) + .map(s -> { + double r = s.value; + ImmutableMap ll = ImmutableMap.builder() + .putAll(Maps.filterKeys(s.labels, + key -> !Objects.equals( + key, le) + )) + .put( + "le", + String.valueOf((long) ((Double.parseDouble(s.labels.get(le))) * scale))) + .build(); + return InternalOps.newSample(s.name, ll, s.timestamp, r); + }) + ).toArray(Sample[]::new) + ); + } + + public SampleFamily histogram_percentile(List percentiles) { + Preconditions.checkArgument(percentiles.size() > 0); + int[] p = percentiles.stream().mapToInt(i -> i).toArray(); + ExpressionParsingContext.get().ifPresent(ctx -> { + Preconditions.checkState( + ctx.isHistogram, "histogram() should be invoked before invoking histogram_percentile()"); + ctx.percentiles = p; + }); + return this; + } + + public SampleFamily service(List labelKeys, Layer layer) { + Preconditions.checkArgument(labelKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE; + ctx.scopeLabels.addAll(labelKeys); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new ServiceEntityDescription(labelKeys, layer, Const.POINT)); + } + + public SampleFamily service(List labelKeys, String delimiter, Layer layer) { + Preconditions.checkArgument(labelKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE; + ctx.scopeLabels.addAll(labelKeys); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new ServiceEntityDescription(labelKeys, layer, delimiter)); + } + + public SampleFamily instance(List serviceKeys, String serviceDelimiter, + List instanceKeys, String instanceDelimiter, + Layer layer, Closure> propertiesExtractor) { + Preconditions.checkArgument(serviceKeys.size() > 0); + Preconditions.checkArgument(instanceKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE_INSTANCE; + ctx.scopeLabels.addAll(serviceKeys); + ctx.scopeLabels.addAll(instanceKeys); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new InstanceEntityDescription( + serviceKeys, instanceKeys, layer, serviceDelimiter, instanceDelimiter, + propertiesExtractor == null ? null : propertiesExtractor::call)); + } + + public SampleFamily instance(List serviceKeys, String serviceDelimiter, + List instanceKeys, String instanceDelimiter, + Layer layer, PropertiesExtractor propertiesExtractor) { + Preconditions.checkArgument(serviceKeys.size() > 0); + Preconditions.checkArgument(instanceKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE_INSTANCE; + ctx.scopeLabels.addAll(serviceKeys); + ctx.scopeLabels.addAll(instanceKeys); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new InstanceEntityDescription( + serviceKeys, instanceKeys, layer, serviceDelimiter, instanceDelimiter, propertiesExtractor)); + } + + public SampleFamily instance(List serviceKeys, List instanceKeys, Layer layer) { + return instance(serviceKeys, Const.POINT, instanceKeys, Const.POINT, layer, (Closure>) null); + } + + public SampleFamily endpoint(List serviceKeys, List endpointKeys, String delimiter, Layer layer) { + Preconditions.checkArgument(serviceKeys.size() > 0); + Preconditions.checkArgument(endpointKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.ENDPOINT; + ctx.scopeLabels.addAll(serviceKeys); + ctx.scopeLabels.addAll(endpointKeys); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new EndpointEntityDescription(serviceKeys, endpointKeys, layer, delimiter)); + } + + public SampleFamily endpoint(List serviceKeys, List endpointKeys, Layer layer) { + return endpoint(serviceKeys, endpointKeys, Const.POINT, layer); + } + + public SampleFamily process(List serviceKeys, List serviceInstanceKeys, List processKeys, String layerKey) { + Preconditions.checkArgument(serviceKeys.size() > 0); + Preconditions.checkArgument(serviceInstanceKeys.size() > 0); + Preconditions.checkArgument(processKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.PROCESS; + ctx.scopeLabels.addAll(serviceKeys); + ctx.scopeLabels.addAll(serviceInstanceKeys); + ctx.scopeLabels.addAll(processKeys); + ctx.scopeLabels.add(layerKey); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new ProcessEntityDescription(serviceKeys, serviceInstanceKeys, processKeys, layerKey, Const.POINT)); + } + + public SampleFamily serviceRelation(DetectPoint detectPoint, List sourceServiceKeys, List destServiceKeys, Layer layer) { + Preconditions.checkArgument(sourceServiceKeys.size() > 0); + Preconditions.checkArgument(destServiceKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE_RELATION; + ctx.scopeLabels.addAll(sourceServiceKeys); + ctx.scopeLabels.addAll(destServiceKeys); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new ServiceRelationEntityDescription(sourceServiceKeys, destServiceKeys, detectPoint, layer, Const.POINT, null)); + } + + public SampleFamily serviceRelation(DetectPoint detectPoint, List sourceServiceKeys, List destServiceKeys, String delimiter, Layer layer, String componentIdKey) { + Preconditions.checkArgument(sourceServiceKeys.size() > 0); + Preconditions.checkArgument(destServiceKeys.size() > 0); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.SERVICE_RELATION; + ctx.scopeLabels.addAll(sourceServiceKeys); + ctx.scopeLabels.addAll(destServiceKeys); + ctx.scopeLabels.add(componentIdKey); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new ServiceRelationEntityDescription(sourceServiceKeys, destServiceKeys, detectPoint, layer, delimiter, componentIdKey)); + } + + public SampleFamily forEach(List array, Closure each) { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build(this.context, Arrays.stream(this.samples).map(sample -> { + Map labels = Maps.newHashMap(sample.getLabels()); + for (String element : array) { + each.call(element, labels); + } + return sample.toBuilder().labels(ImmutableMap.copyOf(labels)).build(); + }).toArray(Sample[]::new)); + } + + public SampleFamily forEach(List array, ForEachFunction each) { + if (this == EMPTY) { + return EMPTY; + } + return SampleFamily.build(this.context, Arrays.stream(this.samples).map(sample -> { + Map labels = Maps.newHashMap(sample.getLabels()); + for (String element : array) { + each.accept(element, labels); + } + return sample.toBuilder().labels(ImmutableMap.copyOf(labels)).build(); + }).toArray(Sample[]::new)); + } + + public SampleFamily processRelation(String detectPointKey, List serviceKeys, List instanceKeys, String sourceProcessIdKey, String destProcessIdKey, String componentKey) { + Preconditions.checkArgument(serviceKeys.size() > 0); + Preconditions.checkArgument(instanceKeys.size() > 0); + Preconditions.checkArgument(StringUtil.isNotEmpty(sourceProcessIdKey)); + Preconditions.checkArgument(StringUtil.isNotEmpty(destProcessIdKey)); + ExpressionParsingContext.get().ifPresent(ctx -> { + ctx.scopeType = ScopeType.PROCESS_RELATION; + ctx.scopeLabels.addAll(serviceKeys); + ctx.scopeLabels.addAll(instanceKeys); + ctx.scopeLabels.add(detectPointKey); + ctx.scopeLabels.add(sourceProcessIdKey); + ctx.scopeLabels.add(destProcessIdKey); + ctx.scopeLabels.add(componentKey); + }); + if (this == EMPTY) { + return EMPTY; + } + return createMeterSamples(new ProcessRelationEntityDescription(serviceKeys, instanceKeys, sourceProcessIdKey, destProcessIdKey, detectPointKey, componentKey, Const.POINT)); + } + + private SampleFamily createMeterSamples(EntityDescription entityDescription) { + Map meterSamples = new HashMap<>(); + Arrays.stream(samples) + .collect(groupingBy(it -> InternalOps.getLabels(entityDescription.getLabelKeys(), it), + mapping(identity(), toList()) + )) + .forEach((labels, samples) -> { + MeterEntity meterEntity = InternalOps.buildMeterEntity(samples, entityDescription); + meterSamples.put( + meterEntity, InternalOps.left(samples, entityDescription.getLabelKeys())); + }); + + this.context.setMeterSamples(meterSamples); + //This samples is original, The grouped samples is in context which mapping with MeterEntity + return SampleFamily.build(this.context, samples); + } + + private SampleFamily match(String[] labels, Function2 op) { + Preconditions.checkArgument(labels.length % 2 == 0); + Map ll = new HashMap<>(labels.length / 2); + for (int i = 0; i < labels.length; i += 2) { + ll.put(labels[i], labels[i + 1]); + } + Sample[] ss = Arrays.stream(samples) + .filter(sample -> ll.entrySet() + .stream() + .allMatch( + entry -> op.apply(sample.labels.getOrDefault(entry.getKey(), ""), + entry.getValue() + ))) + .toArray(Sample[]::new); + return ss.length > 0 ? SampleFamily.build(this.context, ss) : EMPTY; + } + + private SampleFamily valueMatch(CompType compType, + double compValue, + Function3 op) { + Sample[] ss = Arrays.stream(samples) + .filter(sample -> op.apply(compType, sample.value, compValue)).toArray(Sample[]::new); + return ss.length > 0 ? SampleFamily.build(this.context, ss) : EMPTY; + } + + SampleFamily newValue(Function transform) { + if (this == EMPTY) { + return EMPTY; + } + Sample[] ss = new Sample[samples.length]; + for (int i = 0; i < ss.length; i++) { + ss[i] = samples[i].newValue(transform); + } + return SampleFamily.build(this.context, ss); + } + + private SampleFamily newValue(SampleFamily another, Function2 transform) { + Sample[] ss = Arrays.stream(samples) + .flatMap(cs -> io.vavr.collection.Stream.of(another.samples) + .find(as -> cs.labels.equals(as.labels)) + .map(as -> cs.toBuilder() + .value(transform.apply(cs.value, + as.value + ))) + .map(Sample.SampleBuilder::build) + .toJavaStream()) + .toArray(Sample[]::new); + return ss.length > 0 ? SampleFamily.build(this.context, ss) : EMPTY; + } + + public SampleFamily downsampling(final DownsamplingType type) { + ExpressionParsingContext.get().ifPresent(it -> it.downsampling = type); + return this; + } + + /** + * Decorate the service meter entity with the given closure. + */ + public SampleFamily decorate(Closure c) { + ExpressionParsingContext.get().ifPresent(ctx -> { + if (ctx.getScopeType() != ScopeType.SERVICE) { + throw new IllegalStateException("decorate() should be invoked after service()"); + } + if (ctx.isHistogram()) { + throw new IllegalStateException("decorate() not supported for histogram metrics"); + } + }); + if (this == EMPTY) { + return EMPTY; + } + this.context.getMeterSamples().keySet().forEach(meterEntity -> { + // Only service meter entity can be decorated + if (meterEntity.getScopeType().equals(ScopeType.SERVICE)) { + c.call(meterEntity); + } + }); + return this; + } + + public SampleFamily decorate(DecorateFunction c) { + ExpressionParsingContext.get().ifPresent(ctx -> { + if (ctx.getScopeType() != ScopeType.SERVICE) { + throw new IllegalStateException("decorate() should be invoked after service()"); + } + if (ctx.isHistogram()) { + throw new IllegalStateException("decorate() not supported for histogram metrics"); + } + }); + if (this == EMPTY) { + return EMPTY; + } + this.context.getMeterSamples().keySet().forEach(meterEntity -> { + if (meterEntity.getScopeType().equals(ScopeType.SERVICE)) { + c.accept(meterEntity); + } + }); + return this; + } + + /** + * The parsing context holds key results more than sample collection. + */ + @ToString + @EqualsAndHashCode(exclude = "metricName") + @Getter + @Setter + @Builder + public static class RunningContext { + + static RunningContext EMPTY = instance(); + + static RunningContext instance() { + return RunningContext.builder() + .defaultHistogramBucketUnit(TimeUnit.SECONDS) + .build(); + } + + private String metricName; + + @Builder.Default + private Map meterSamples = new HashMap<>(); + + private TimeUnit defaultHistogramBucketUnit; + } + + private static class InternalOps { + + private static Sample[] left(List samples, List labelKeys) { + return samples.stream().map(s -> { + ImmutableMap ll = ImmutableMap.builder() + .putAll(Maps.filterKeys(s.labels, + key -> !labelKeys.contains(key) + )) + .build(); + return s.toBuilder().labels(ll).build(); + }).toArray(Sample[]::new); + } + + private static String dim(List samples, List labelKeys, String delimiter) { + String name = labelKeys.stream() + .map(k -> samples.get(0).labels.getOrDefault(k, "")) + .filter(v -> !StringUtil.isEmpty(v)) + .collect(Collectors.joining(StringUtil.isEmpty(delimiter) ? Const.POINT : delimiter)); + return name; + } + + private static MeterEntity buildMeterEntity(List samples, + EntityDescription entityDescription) { + switch (entityDescription.getScopeType()) { + case SERVICE: + ServiceEntityDescription serviceEntityDescription = (ServiceEntityDescription) entityDescription; + return MeterEntity.newService( + InternalOps.dim(samples, serviceEntityDescription.getServiceKeys(), serviceEntityDescription.getDelimiter()), + serviceEntityDescription.getLayer() + ); + case SERVICE_INSTANCE: + InstanceEntityDescription instanceEntityDescription = (InstanceEntityDescription) entityDescription; + Map properties = null; + if (instanceEntityDescription.getPropertiesExtractor() != null) { + properties = instanceEntityDescription.getPropertiesExtractor().apply(samples.get(0).labels); + } + return MeterEntity.newServiceInstance( + InternalOps.dim(samples, instanceEntityDescription.getServiceKeys(), instanceEntityDescription.getServiceDelimiter()), + InternalOps.dim(samples, instanceEntityDescription.getInstanceKeys(), instanceEntityDescription.getInstanceDelimiter()), + instanceEntityDescription.getLayer(), + properties + ); + case ENDPOINT: + EndpointEntityDescription endpointEntityDescription = (EndpointEntityDescription) entityDescription; + return MeterEntity.newEndpoint( + InternalOps.dim(samples, endpointEntityDescription.getServiceKeys(), endpointEntityDescription.getDelimiter()), + InternalOps.dim(samples, endpointEntityDescription.getEndpointKeys(), endpointEntityDescription.getDelimiter()), + endpointEntityDescription.getLayer() + ); + case PROCESS: + final ProcessEntityDescription processEntityDescription = (ProcessEntityDescription) entityDescription; + return MeterEntity.newProcess( + InternalOps.dim(samples, processEntityDescription.getServiceKeys(), processEntityDescription.getDelimiter()), + InternalOps.dim(samples, processEntityDescription.getServiceInstanceKeys(), processEntityDescription.getDelimiter()), + InternalOps.dim(samples, processEntityDescription.getProcessKeys(), processEntityDescription.getDelimiter()), + InternalOps.dim(samples, List.of(processEntityDescription.getLayerKey()), processEntityDescription.getDelimiter()) + ); + case SERVICE_RELATION: + ServiceRelationEntityDescription serviceRelationEntityDescription = (ServiceRelationEntityDescription) entityDescription; + final String serviceRelationComponentValue = InternalOps.dim(samples, + Collections.singletonList(serviceRelationEntityDescription.getComponentIdKey()), serviceRelationEntityDescription.getDelimiter()); + int serviceRelationComponentId = StringUtil.isNotEmpty(serviceRelationComponentValue) ? Integer.parseInt(serviceRelationComponentValue) : 0; + return MeterEntity.newServiceRelation( + InternalOps.dim(samples, serviceRelationEntityDescription.getSourceServiceKeys(), serviceRelationEntityDescription.getDelimiter()), + InternalOps.dim(samples, serviceRelationEntityDescription.getDestServiceKeys(), serviceRelationEntityDescription.getDelimiter()), + serviceRelationEntityDescription.getDetectPoint(), serviceRelationEntityDescription.getLayer(), serviceRelationComponentId + ); + case PROCESS_RELATION: + final ProcessRelationEntityDescription processRelationEntityDescription = (ProcessRelationEntityDescription) entityDescription; + final String detectPointValue = InternalOps.dim(samples, Collections.singletonList(processRelationEntityDescription.getDetectPointKey()), processRelationEntityDescription.getDelimiter()); + DetectPoint point = StringUtils.equalsAnyIgnoreCase(detectPointValue, "server") ? DetectPoint.SERVER : DetectPoint.CLIENT; + final String componentValue = InternalOps.dim(samples, Collections.singletonList(processRelationEntityDescription.getComponentKey()), processRelationEntityDescription.getDelimiter()); + final int componentId = StringUtil.isNotEmpty(componentValue) ? Integer.parseInt(componentValue) : 0; + return MeterEntity.newProcessRelation( + InternalOps.dim(samples, processRelationEntityDescription.getServiceKeys(), processRelationEntityDescription.getDelimiter()), + InternalOps.dim(samples, processRelationEntityDescription.getInstanceKeys(), processRelationEntityDescription.getDelimiter()), + InternalOps.dim(samples, Collections.singletonList(processRelationEntityDescription.getSourceProcessIdKey()), processRelationEntityDescription.getDelimiter()), + InternalOps.dim(samples, Collections.singletonList(processRelationEntityDescription.getDestProcessIdKey()), processRelationEntityDescription.getDelimiter()), + componentId, + point + ); + default: + throw new UnexpectedException( + "Unexpected scope type of entityDescription " + entityDescription); + } + } + + private static Sample newSample(String name, + ImmutableMap labels, + long timestamp, + double newValue) { + return Sample.builder() + .value(newValue) + .labels(labels) + .timestamp(timestamp) + .name(name) + .build(); + } + + private static boolean stringComp(String a, String b) { + if (Strings.isNullOrEmpty(a) && Strings.isNullOrEmpty(b)) { + return true; + } + if (Strings.isNullOrEmpty(a)) { + return false; + } + return a.equals(b); + } + + private static boolean doubleComp(CompType compType, double a, double b) { + int result = Double.compare(a, b); + switch (compType) { + case EQUAL: + return result == 0; + case NOT_EQUAL: + return result != 0; + case GREATER: + return result == 1; + case GREATER_EQUAL: + return result == 0 || result == 1; + case LESS: + return result == -1; + case LESS_EQUAL: + return result == 0 || result == -1; + } + + return false; + } + + private static ImmutableMap getLabels(final List labelKeys, final Sample sample) { + return labelKeys.stream() + .collect(toImmutableMap( + Function.identity(), + labelKey -> sample.labels.getOrDefault(labelKey, "") + )); + } + + private static ImmutableMap groupByExcludedLabel(final String excludedLabelKey, final Sample sample) { + return sample + .labels + .entrySet() + .stream() + .filter(v -> !v.getKey().equals(excludedLabelKey)) + .collect(toImmutableMap(Map.Entry::getKey, Map.Entry::getValue)); + } + } + + private enum CompType { + EQUAL, NOT_EQUAL, LESS, LESS_EQUAL, GREATER, GREATER_EQUAL + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java new file mode 100644 index 000000000000..2893ad6a510d --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.concurrent.TimeUnit; + +/** + * Help to build the {@link SampleFamily}. + */ +public class SampleFamilyBuilder { + private final Sample[] samples; + private final SampleFamily.RunningContext context; + + SampleFamilyBuilder(Sample[] samples, SampleFamily.RunningContext context) { + this.samples = samples; + this.context = context; + } + + public static SampleFamilyBuilder newBuilder(Sample... samples) { + return new SampleFamilyBuilder(samples, SampleFamily.RunningContext.instance()); + } + + public SampleFamilyBuilder defaultHistogramBucketUnit(TimeUnit unit) { + this.context.setDefaultHistogramBucketUnit(unit); + return this; + } + + /** + * Build Sample Family + */ + public SampleFamily build() { + return SampleFamily.build(this.context, this.samples); + } + +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java new file mode 100644 index 000000000000..1c9747b56a25 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl; + +import java.util.Map; +import java.util.function.Consumer; +import java.util.function.Function; +import java.util.function.Predicate; +import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity; + +/** + * Pure Java functional interfaces replacing Groovy Closure parameters in SampleFamily methods. + */ +public final class SampleFamilyFunctions { + + private SampleFamilyFunctions() { + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#tag(groovy.lang.Closure)}. + * Receives a mutable label map and returns the (possibly modified) map. + */ + @FunctionalInterface + public interface TagFunction extends Function, Map> { + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#filter(groovy.lang.Closure)}. + * Tests whether a sample's labels match the filter criteria. + */ + @FunctionalInterface + public interface SampleFilter extends Predicate> { + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#forEach(java.util.List, groovy.lang.Closure)}. + * Called for each element in the array with the element value and a mutable labels map. + */ + @FunctionalInterface + public interface ForEachFunction { + void accept(String element, Map tags); + } + + /** + * Replaces {@code Closure} in {@link SampleFamily#decorate(groovy.lang.Closure)}. + * Decorates service meter entities. + */ + @FunctionalInterface + public interface DecorateFunction extends Consumer { + } + + /** + * Replaces {@code Closure>} in + * {@link SampleFamily#instance(java.util.List, String, java.util.List, String, + * org.apache.skywalking.oap.server.core.analysis.Layer, groovy.lang.Closure)}. + * Extracts instance properties from sample labels. + */ + @FunctionalInterface + public interface PropertiesExtractor extends Function, Map> { + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java new file mode 100644 index 000000000000..5e7e6039ee51 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.counter; + +import com.google.common.collect.ImmutableMap; +import io.vavr.Tuple; +import io.vavr.Tuple2; +import java.util.Map; +import java.util.PriorityQueue; +import java.util.Queue; +import java.util.concurrent.ConcurrentHashMap; +import lombok.AccessLevel; +import lombok.EqualsAndHashCode; +import lombok.RequiredArgsConstructor; +import lombok.ToString; + +/** + * CounterWindow stores a series of counter samples in order to calculate the increase + * or instant rate of increase. + * + */ +@RequiredArgsConstructor(access = AccessLevel.PRIVATE) +@ToString +@EqualsAndHashCode +public class CounterWindow { + + public static final CounterWindow INSTANCE = new CounterWindow(); + + private final Map> lastElementMap = new ConcurrentHashMap<>(); + private final Map>> windows = new ConcurrentHashMap<>(); + + public Tuple2 increase(String name, ImmutableMap labels, Double value, long windowSize, long now) { + ID id = new ID(name, labels); + Queue> window = windows.computeIfAbsent(id, unused -> new PriorityQueue<>()); + synchronized (window) { + window.offer(Tuple.of(now, value)); + long waterLevel = now - windowSize; + Tuple2 peek = window.peek(); + if (peek._1 > waterLevel) { + return peek; + } + + Tuple2 result = peek; + while (peek._1 < waterLevel) { + result = window.poll(); + peek = window.element(); + } + + // Choose the closed slot to the expected timestamp + if (waterLevel - result._1 <= peek._1 - waterLevel) { + return result; + } + + return peek; + } + } + + public Tuple2 pop(String name, ImmutableMap labels, Double value, long now) { + ID id = new ID(name, labels); + + Tuple2 element = Tuple.of(now, value); + Tuple2 result = lastElementMap.put(id, element); + if (result == null) { + return element; + } + return result; + } + + public void reset() { + windows.clear(); + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java new file mode 100644 index 000000000000..45ce13c0f505 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.counter; + +import com.google.common.collect.ImmutableMap; +import lombok.EqualsAndHashCode; +import lombok.RequiredArgsConstructor; +import lombok.ToString; + +@RequiredArgsConstructor +@EqualsAndHashCode +@ToString +class ID { + + private final String name; + + private final ImmutableMap labels; +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java new file mode 100644 index 000000000000..47759ec20c07 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.registry; + +import org.apache.commons.lang3.StringUtils; +import org.apache.skywalking.library.kubernetes.ObjectID; +import org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry; +import org.apache.skywalking.oap.server.core.Const; +import org.apache.skywalking.oap.server.core.analysis.DownSampling; +import org.apache.skywalking.oap.server.core.analysis.IDManager; +import org.apache.skywalking.oap.server.core.analysis.TimeBucket; +import org.apache.skywalking.oap.server.core.analysis.manual.process.ProcessDetectType; +import org.apache.skywalking.oap.server.core.analysis.manual.process.ProcessTraffic; +import org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor; + +/** + * The dynamic entity registry for {@link ProcessTraffic} + */ +public class ProcessRegistry { + + public static final String LOCAL_VIRTUAL_PROCESS = "UNKNOWN_LOCAL"; + public static final String REMOTE_VIRTUAL_PROCESS = "UNKNOWN_REMOTE"; + + /** + * Generate virtual local process under the instance + * @return the process id + */ + public static String generateVirtualLocalProcess(String service, String instance) { + return generateVirtualProcess(service, instance, LOCAL_VIRTUAL_PROCESS); + } + + /** + * Generate virtual remote process under the instance + * trying to generate the name in the kubernetes environment through the remote address + * @return the process id + */ + public static String generateVirtualRemoteProcess(String service, String instance, String remoteAddress) { + // remove port + String ip = StringUtils.substringBeforeLast(remoteAddress, ":"); + + // find remote through k8s metadata + ObjectID metadata = K8sInfoRegistry.getInstance().findPodByIP(ip); + if (metadata == ObjectID.EMPTY) { + metadata = K8sInfoRegistry.getInstance().findServiceByIP(ip); + } + String name = metadata.toString(); + // if not exists, then just use remote unknown + if (StringUtils.isBlank(name)) { + name = REMOTE_VIRTUAL_PROCESS; + } + + return generateVirtualProcess(service, instance, name); + } + + public static String generateVirtualProcess(String service, String instance, String processName) { + final ProcessTraffic traffic = new ProcessTraffic(); + final String serviceId = IDManager.ServiceID.buildId(service, true); + traffic.setServiceId(serviceId); + traffic.setInstanceId(IDManager.ServiceInstanceID.buildId(serviceId, instance)); + traffic.setName(processName); + traffic.setAgentId(Const.EMPTY_STRING); + traffic.setLabelsJson(Const.EMPTY_STRING); + traffic.setDetectType(ProcessDetectType.VIRTUAL.value()); + final long timeBucket = TimeBucket.getTimeBucket(System.currentTimeMillis(), DownSampling.Minute); + traffic.setTimeBucket(timeBucket); + traffic.setLastPingTimestamp(timeBucket); + MetricsStreamProcessor.getInstance().in(traffic); + return traffic.id().build(); + } +} \ No newline at end of file diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java new file mode 100644 index 000000000000..53d9ff5fe591 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt; + +import com.google.common.base.Strings; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.Maps; +import java.util.Arrays; +import java.util.Map; +import org.apache.skywalking.oap.meter.analyzer.dsl.Sample; +import org.apache.skywalking.oap.meter.analyzer.k8s.K8sInfoRegistry; + +public enum K8sRetagType implements Retag { + Pod2Service { + @Override + public Sample[] execute(final Sample[] ss, + final String newLabelName, + final String existingLabelName, + final String namespaceLabelName) { + return Arrays.stream(ss).map(sample -> { + String podName = sample.getLabels().get(existingLabelName); + String namespace = sample.getLabels().get(namespaceLabelName); + if (!Strings.isNullOrEmpty(podName) && !Strings.isNullOrEmpty(namespace)) { + String serviceName = K8sInfoRegistry.getInstance().findServiceName(namespace, podName); + if (Strings.isNullOrEmpty(serviceName)) { + serviceName = BLANK; + } + Map labels = Maps.newHashMap(sample.getLabels()); + labels.put(newLabelName, serviceName); + return sample.toBuilder().labels(ImmutableMap.copyOf(labels)).build(); + } + return sample; + }).toArray(Sample[]::new); + } + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java new file mode 100644 index 000000000000..d95930f1a808 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java @@ -0,0 +1,27 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.dsl.tagOpt; + +import org.apache.skywalking.oap.meter.analyzer.dsl.Sample; + +public interface Retag { + String BLANK = ""; + + Sample[] execute(Sample[] ss, String newLabelName, String existingLabelName, String namespaceLabelName); +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java new file mode 100644 index 000000000000..c08e10b62ea9 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.k8s; + +import com.google.common.cache.CacheBuilder; +import com.google.common.cache.CacheLoader; +import com.google.common.cache.LoadingCache; +import io.fabric8.kubernetes.api.model.Pod; +import io.fabric8.kubernetes.api.model.Service; +import lombok.SneakyThrows; +import org.apache.skywalking.library.kubernetes.KubernetesPods; +import org.apache.skywalking.library.kubernetes.KubernetesServices; +import org.apache.skywalking.library.kubernetes.ObjectID; + +import java.time.Duration; +import java.util.Collection; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; + +import static java.util.Objects.requireNonNull; + +public class K8sInfoRegistry { + private final static K8sInfoRegistry INSTANCE = new K8sInfoRegistry(); + private final LoadingCache podServiceMap; + private final LoadingCache ipPodMap; + private final LoadingCache ipServiceMap; + + private K8sInfoRegistry() { + ipPodMap = CacheBuilder.newBuilder() + .expireAfterWrite(Duration.ofMinutes(3)) + .build(CacheLoader.from(ip -> KubernetesPods.INSTANCE + .findByIP(ip) + .map(it -> ObjectID + .builder() + .name(it.getMetadata().getName()) + .namespace(it.getMetadata().getNamespace()) + .build()) + .orElse(ObjectID.EMPTY))); + + ipServiceMap = CacheBuilder.newBuilder() + .expireAfterWrite(Duration.ofMinutes(3)) + .build(CacheLoader.from(ip -> KubernetesServices.INSTANCE + .list() + .stream() + .filter(it -> it.getSpec() != null) + .filter(it -> it.getStatus() != null) + .filter(it -> it.getMetadata() != null) + .filter(it -> (it.getSpec().getClusterIPs() != null && + it.getSpec().getClusterIPs().stream() + .anyMatch(clusterIP -> Objects.equals(clusterIP, ip))) + || (it.getStatus().getLoadBalancer() != null && + it.getStatus().getLoadBalancer().getIngress() != null && + it.getStatus().getLoadBalancer().getIngress().stream() + .anyMatch(ingress -> Objects.equals(ingress.getIp(), ip)))) + .map(it -> ObjectID + .builder() + .name(it.getMetadata().getName()) + .namespace(it.getMetadata().getNamespace()) + .build()) + .findFirst() + .orElse(ObjectID.EMPTY))); + + podServiceMap = CacheBuilder.newBuilder() + .expireAfterWrite(Duration.ofMinutes(3)) + .build(CacheLoader.from(podObjectID -> { + final Optional pod = KubernetesPods.INSTANCE + .findByObjectID( + ObjectID + .builder() + .name(podObjectID.name()) + .namespace(podObjectID.namespace()) + .build()); + + if (!pod.isPresent() + || pod.get().getMetadata() == null + || pod.get().getMetadata().getLabels() == null) { + return ObjectID.EMPTY; + } + + final Optional service = KubernetesServices.INSTANCE + .list() + .stream() + .filter(it -> it.getMetadata() != null) + .filter(it -> Objects.equals(it.getMetadata().getNamespace(), pod.get().getMetadata().getNamespace())) + .filter(it -> it.getSpec() != null) + .filter(it -> requireNonNull(it.getSpec()).getSelector() != null) + .filter(it -> !it.getSpec().getSelector().isEmpty()) + .filter(it -> { + final Map labels = pod.get().getMetadata().getLabels(); + final Map selector = it.getSpec().getSelector(); + return hasIntersection(selector.entrySet(), labels.entrySet()); + }) + .findFirst(); + if (!service.isPresent()) { + return ObjectID.EMPTY; + } + return ObjectID + .builder() + .name(service.get().getMetadata().getName()) + .namespace(service.get().getMetadata().getNamespace()) + .build(); + })); + } + + public static K8sInfoRegistry getInstance() { + return INSTANCE; + } + + @SneakyThrows + public String findServiceName(String namespace, String podName) { + return findService(namespace, podName).toString(); + } + + @SneakyThrows + public ObjectID findService(String namespace, String podName) { + return this.podServiceMap.get( + ObjectID + .builder() + .name(podName) + .namespace(namespace) + .build()); + } + + @SneakyThrows + public ObjectID findPodByIP(String ip) { + return this.ipPodMap.get(ip); + } + + @SneakyThrows + public ObjectID findServiceByIP(String ip) { + return this.ipServiceMap.get(ip); + } + + private boolean hasIntersection(Collection o, Collection c) { + Objects.requireNonNull(o); + Objects.requireNonNull(c); + for (final Object value : o) { + if (!c.contains(value)) { + return false; + } + } + return true; + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java new file mode 100644 index 000000000000..4dc5ab4c9c8a --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.prometheus; + +import com.google.common.cache.CacheBuilder; +import com.google.common.cache.CacheLoader; +import com.google.common.cache.LoadingCache; +import com.google.common.collect.ImmutableMap; +import io.vavr.Tuple; +import io.vavr.Tuple2; +import java.util.Collections; +import java.util.Optional; +import java.util.concurrent.ExecutionException; +import java.util.regex.Pattern; +import java.util.stream.Stream; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.meter.analyzer.dsl.Sample; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily; +import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamilyBuilder; +import org.apache.skywalking.oap.server.library.util.prometheus.metrics.Counter; +import org.apache.skywalking.oap.server.library.util.prometheus.metrics.Gauge; +import org.apache.skywalking.oap.server.library.util.prometheus.metrics.Histogram; +import org.apache.skywalking.oap.server.library.util.prometheus.metrics.Metric; +import org.apache.skywalking.oap.server.library.util.prometheus.metrics.Summary; + +import static com.google.common.collect.ImmutableMap.toImmutableMap; +import static io.vavr.API.$; +import static io.vavr.API.Case; +import static io.vavr.API.Match; +import static io.vavr.Predicates.instanceOf; +import static java.util.stream.Collectors.toList; +import static org.apache.skywalking.oap.meter.analyzer.Analyzer.NIL; + +/** + * PrometheusMetricConverter converts prometheus metrics to meter-system metrics, then store them to backend storage. + */ +@Slf4j +public class PrometheusMetricConverter { + private static final Pattern METRICS_NAME_ESCAPE_PATTERN = Pattern.compile("[/.]"); + + private static final LoadingCache ESCAPED_METRICS_NAME_CACHE = + CacheBuilder.newBuilder() + .maximumSize(1000) + .build(new CacheLoader() { + @Override + public String load(final String name) { + return METRICS_NAME_ESCAPE_PATTERN.matcher(name).replaceAll("_"); + } + }); + + public static ImmutableMap convertPromMetricToSampleFamily(Stream metricStream) { + return metricStream + .peek(metric -> log.debug("Prom metric to be convert to SampleFamily: {}", metric)) + .flatMap(PrometheusMetricConverter::convertMetric) + .filter(t -> t != NIL && t._2.samples.length > 0) + .peek(t -> log.debug("SampleFamily: {}", t)) + .collect(toImmutableMap(Tuple2::_1, Tuple2::_2, (a, b) -> { + log.debug("merge {} {}", a, b); + Sample[] m = new Sample[a.samples.length + b.samples.length]; + System.arraycopy(a.samples, 0, m, 0, a.samples.length); + System.arraycopy(b.samples, 0, m, a.samples.length, b.samples.length); + return SampleFamilyBuilder.newBuilder(m).build(); + })); + } + + private static Stream> convertMetric(Metric metric) { + return Match(metric).of( + Case($(instanceOf(Histogram.class)), t -> Stream.of( + Tuple.of(escapedName(metric.getName() + "_count"), SampleFamilyBuilder.newBuilder(Sample.builder().name(escapedName(metric.getName() + "_count")) + .timestamp(metric.getTimestamp()).labels(ImmutableMap.copyOf(metric.getLabels())).value(((Histogram) metric).getSampleCount()).build()).build()), + Tuple.of(escapedName(metric.getName() + "_sum"), SampleFamilyBuilder.newBuilder(Sample.builder().name(escapedName(metric.getName() + "_sum")) + .timestamp(metric.getTimestamp()).labels(ImmutableMap.copyOf(metric.getLabels())).value(((Histogram) metric).getSampleSum()).build()).build()), + convertToSample(metric).orElse(NIL))), + Case($(instanceOf(Summary.class)), t -> Stream.of( + Tuple.of(escapedName(metric.getName() + "_count"), SampleFamilyBuilder.newBuilder(Sample.builder().name(escapedName(metric.getName() + "_count")) + .timestamp(metric.getTimestamp()).labels(ImmutableMap.copyOf(metric.getLabels())).value(((Summary) metric).getSampleCount()).build()).build()), + Tuple.of(escapedName(metric.getName() + "_sum"), SampleFamilyBuilder.newBuilder(Sample.builder().name(escapedName(metric.getName() + "_sum")) + .timestamp(metric.getTimestamp()).labels(ImmutableMap.copyOf(metric.getLabels())).value(((Summary) metric).getSampleSum()).build()).build()), + convertToSample(metric).orElse(NIL))), + Case($(), t -> Stream.of(convertToSample(metric).orElse(NIL))) + ); + } + + private static Optional> convertToSample(Metric metric) { + Sample[] ss = Match(metric).of( + Case($(instanceOf(Counter.class)), t -> Collections.singletonList(Sample.builder() + .name(escapedName(t.getName())) + .labels(ImmutableMap.copyOf(t.getLabels())) + .timestamp(t.getTimestamp()) + .value(t.getValue()) + .build())), + Case($(instanceOf(Gauge.class)), t -> Collections.singletonList(Sample.builder() + .name(escapedName(t.getName())) + .labels(ImmutableMap.copyOf(t.getLabels())) + .timestamp(t.getTimestamp()) + .value(t.getValue()) + .build())), + Case($(instanceOf(Histogram.class)), t -> t.getBuckets() + .entrySet().stream() + .map(b -> Sample.builder() + .name(escapedName(t.getName())) + .labels(ImmutableMap.builder() + .putAll(t.getLabels()) + .put("le", b.getKey().toString()) + .build()) + .timestamp(t.getTimestamp()) + .value(b.getValue()) + .build()).collect(toList())), + Case($(instanceOf(Summary.class)), + t -> t.getQuantiles().entrySet().stream() + .map(b -> Sample.builder() + .name(escapedName(t.getName())) + .labels(ImmutableMap.builder() + .putAll(t.getLabels()) + .put("quantile", b.getKey().toString()) + .build()) + .timestamp(t.getTimestamp()) + .value(b.getValue()) + .build()).collect(toList())) + ).toArray(new Sample[0]); + if (ss.length < 1) { + return Optional.empty(); + } + return Optional.of(Tuple.of(escapedName(metric.getName()), SampleFamilyBuilder.newBuilder(ss).build())); + } + + // Returns the escaped name of the given one, with "." and "/" replaced by "_" + protected static String escapedName(final String name) { + try { + return ESCAPED_METRICS_NAME_CACHE.get(name); + } catch (ExecutionException e) { + log.error("Failed to get escaped metrics name from cache", e); + return METRICS_NAME_ESCAPE_PATTERN.matcher(name).replaceAll("_"); + } + } +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java new file mode 100644 index 000000000000..e2f2ff44abf7 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java @@ -0,0 +1,37 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.prometheus.rule; + +import lombok.AllArgsConstructor; +import lombok.Builder; +import lombok.Data; +import lombok.NoArgsConstructor; +import org.apache.skywalking.oap.meter.analyzer.MetricRuleConfig; + +/** + * MetricsRule holds the parsing expression. + */ +@Data +@Builder +@NoArgsConstructor +@AllArgsConstructor +public class MetricsRule implements MetricRuleConfig.RuleConfig { + private String name; + private String exp; +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java new file mode 100644 index 000000000000..47cca3896ab3 --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java @@ -0,0 +1,40 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.prometheus.rule; + +import lombok.Data; +import lombok.NoArgsConstructor; +import org.apache.skywalking.oap.meter.analyzer.MetricRuleConfig; + +import java.util.List; + +/** + * Rule contains the global configuration of prometheus fetcher. + */ +@Data +@NoArgsConstructor +public class Rule implements MetricRuleConfig { + private String name; + private String metricPrefix; + private String expSuffix; + private String expPrefix; + private String filter; + private String initExp; + private List metricsRules; +} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java new file mode 100644 index 000000000000..d9d8f3401bbf --- /dev/null +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.meter.analyzer.prometheus.rule; + +import java.io.File; + +import java.io.FileReader; +import java.io.IOException; +import java.io.Reader; + +import java.nio.file.FileSystems; +import java.nio.file.Files; +import java.nio.file.Path; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.stream.Collectors; + +import java.util.stream.Stream; + +import org.apache.skywalking.oap.server.core.UnexpectedException; +import org.apache.skywalking.oap.server.library.util.ResourceUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.yaml.snakeyaml.Yaml; + +/** + * Rules is factory to instance {@link Rule} from a local file. + */ +public class Rules { + private static final Logger LOG = LoggerFactory.getLogger(Rule.class); + + public static List loadRules(final String path) throws IOException { + return loadRules(path, Collections.emptyList()); + } + + public static List loadRules(final String path, List enabledRules) throws IOException { + + final Path root = ResourceUtils.getPath(path); + + Map formedEnabledRules = enabledRules + .stream() + .map(rule -> { + rule = rule.trim(); + if (rule.startsWith("/")) { + rule = rule.substring(1); + } + if (!rule.endsWith(".yaml") && !rule.endsWith(".yml")) { + return rule + "{.yaml,.yml}"; + } + return rule; + }) + .collect(Collectors.toMap(rule -> rule, $ -> false)); + List rules; + try (Stream stream = Files.walk(root)) { + rules = stream + .filter(it -> formedEnabledRules.keySet().stream() + .anyMatch(rule -> { + boolean matches = FileSystems.getDefault().getPathMatcher("glob:" + rule) + .matches(root.relativize(it)); + if (matches) { + formedEnabledRules.put(rule, true); + } + return matches; + })) + .map(pathPointer -> { + // Use relativized file path without suffix as the rule name. + String relativizePath = root.relativize(pathPointer).toString(); + String ruleName = relativizePath.substring(0, relativizePath.lastIndexOf(".")); + return getRulesFromFile(ruleName, pathPointer); + }) + .filter(Objects::nonNull) + .collect(Collectors.toList()) ; + } + + if (formedEnabledRules.containsValue(false)) { + List rulesNotFound = formedEnabledRules.keySet().stream() + .filter(rule -> !formedEnabledRules.get(rule)) + .collect(Collectors.toList()); + throw new UnexpectedException("Some configuration files of enabled rules are not found, enabled rules: " + rulesNotFound); + } + return rules; + } + + private static Rule getRulesFromFile(String ruleName, Path path) { + File file = path.toFile(); + if (!file.isFile() || file.isHidden()) { + return null; + } + try (Reader r = new FileReader(file)) { + Rule rule = new Yaml().loadAs(r, Rule.class); + if (rule == null) { + return null; + } + rule.setName(ruleName); + return rule; + } catch (IOException e) { + throw new UnexpectedException("Load rule file" + file.getName() + " failed", e); + } + } +} diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java b/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java rename to test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/test/script-compiler/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/illegal-yaml/test.yml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/illegal-yaml/test.yml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/single-file-case.yaml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/single-file-case.yaml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/case1.yaml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/case1.yaml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/case2.yml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/case2.yml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/case3.yaml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/case3.yaml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml diff --git a/oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/empty.yaml b/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml similarity index 100% rename from oap-server/analyzer/meter-analyzer/src/test/resources/otel-rules/test-folder/empty.yaml rename to test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml diff --git a/oap-server/analyzer/hierarchy-v2/pom.xml b/test/script-compiler/pom.xml similarity index 73% rename from oap-server/analyzer/hierarchy-v2/pom.xml rename to test/script-compiler/pom.xml index 42e49678b2e6..24d1865c4783 100644 --- a/oap-server/analyzer/hierarchy-v2/pom.xml +++ b/test/script-compiler/pom.xml @@ -19,20 +19,21 @@ - analyzer + oap-server org.apache.skywalking ${revision} + ../../oap-server/pom.xml 4.0.0 - hierarchy-v2 - Pure Java hierarchy rule provider with static rule registry (no Groovy) + script-compiler + pom - - - org.apache.skywalking - server-core - ${project.version} - - + + mal-v1-with-groovy + lal-v1-with-groovy + hierarchy-v1-with-groovy + mal-lal-v1-v2-checker + hierarchy-v1-v2-checker + From 85b9fb0eaa743026540ad30eca70bb66723025ea Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 1 Mar 2026 10:05:33 +0800 Subject: [PATCH 09/64] Fix MAL/LAL/Hierarchy compiler gaps for full v1-v2 parity - Fix MAL sample collection regression: skip downsampling() method arguments to prevent enum values (MAX, SUM, MIN) from being collected as sample names - Fix MAL safe navigation (?.): parser now correctly propagates safeNav flag to chain segments; code generator uses local StringBuilder to avoid corrupting parent buffer - Fix MAL filter grammar: add closureCondition alternatives to closureBody rule for bare conditions like { tags -> tags.x == 'v' } - Fix MAL downsampling detection for bare identifiers parsed as ExprArgument wrapping MetricExpr - Fix MAL sample ordering: use LinkedHashSet for consistent order - Fix LAL tag() function call: add functionName rule allowing TAG token in functionInvocation for if(tag("LOG_KIND") == ...) patterns - Fix LAL ProcessRegistry support: add PROCESS_REGISTRY to valueAccessPrimary grammar rule - Fix LAL tag statement code generation: wrap single tag entries in Collections.singletonMap() since ExtractorSpec.tag() accepts Map - Fix LAL makeComparison to handle CondFunctionCallContext properly - Add debug logging to all three code generators (MAL, LAL, Hierarchy) showing AST and generated Java source at DEBUG level - Add generateFilterSource() to MALClassGenerator for testing - Add error handling unit tests with demo error comments for MAL (5), LAL (4), and Hierarchy (4) generators - All 1248 checker tests pass: MAL 1187, Filter 29, LAL 10, Hierarchy 22 Co-Authored-By: Claude Opus 4.6 --- oap-server/analyzer/hierarchy/CLAUDE.md | 11 +- .../CompiledHierarchyRuleProvider.java | 68 ++++++ .../compiler/HierarchyRuleClassGenerator.java | 12 +- .../rule}/rt/HierarchyRulePackageHolder.java | 2 +- ...chyDefinitionService$HierarchyRuleProvider | 1 + .../HierarchyRuleClassGeneratorTest.java | 36 +++ .../skywalking/lal/rt/grammar/LALParser.g4 | 8 +- .../analyzer/compiler/LALClassGenerator.java | 46 +++- .../analyzer/compiler/LALScriptParser.java | 20 +- .../oap/log/analyzer/dsl/Binding.java | 19 +- .../skywalking/oap/log/analyzer/dsl/DSL.java | 115 +++------- .../oap/log/analyzer/dsl/LalExpression.java | 9 +- .../analyzer/dsl/spec/filter/FilterSpec.java | 21 ++ .../analyzer/provider/log/LogAnalyzer.java | 19 +- .../log/listener/LogFilterListener.java | 23 ++ .../compiler/LALClassGeneratorTest.java | 33 +++ .../oap/log/analyzer/dsl/DSLV2Test.java | 39 ++-- .../skywalking/mal/rt/grammar/MALParser.g4 | 11 +- .../oap/meter/analyzer/Analyzer.java | 40 +++- .../oap/meter/analyzer/MetricConvert.java | 33 ++- .../analyzer/compiler/MALClassGenerator.java | 210 ++++++++++++++++-- .../analyzer/compiler/MALExpressionModel.java | 2 +- .../analyzer/compiler/MALScriptParser.java | 53 ++++- .../oap/meter/analyzer/dsl/Expression.java | 12 +- .../meter/analyzer/dsl/FilterExpression.java | 63 +----- .../oap/meter/analyzer/dsl/MalExpression.java | 1 - .../oap/meter/analyzer/dsl/MalFilter.java | 3 +- .../analyzer/dsl/SampleFamilyFunctions.java | 9 +- .../compiler/MALClassGeneratorTest.java | 54 +++++ oap-server/server-core/pom.xml | 4 - .../config/HierarchyDefinitionService.java | 85 +++---- .../core/hierarchy/HierarchyService.java | 27 +++ .../config/HierarchyRuleComparisonTest.java | 3 +- .../server/checker/lal/LalComparisonTest.java | 73 ++---- .../server/checker/mal/MalComparisonTest.java | 110 +++------ .../checker/mal/MalFilterComparisonTest.java | 61 ++--- .../oap/meter/analyzer/dsl/MalExpression.java | 10 +- 37 files changed, 878 insertions(+), 468 deletions(-) create mode 100644 oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/CompiledHierarchyRuleProvider.java rename oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/{ => hierarchy/rule}/rt/HierarchyRulePackageHolder.java (92%) create mode 100644 oap-server/analyzer/hierarchy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.core.config.HierarchyDefinitionService$HierarchyRuleProvider diff --git a/oap-server/analyzer/hierarchy/CLAUDE.md b/oap-server/analyzer/hierarchy/CLAUDE.md index f2bf5d1d9a38..4038086264c1 100644 --- a/oap-server/analyzer/hierarchy/CLAUDE.md +++ b/oap-server/analyzer/hierarchy/CLAUDE.md @@ -34,9 +34,13 @@ oap-server/analyzer/hierarchy/ HierarchyRuleScriptParser.java — ANTLR4 facade: expression → AST HierarchyRuleModel.java — Immutable AST model classes HierarchyRuleClassGenerator.java — Javassist code generator - rt/ + CompiledHierarchyRuleProvider.java — SPI provider: compiles rule expressions + hierarchy/rule/rt/ HierarchyRulePackageHolder.java — Class loading anchor (empty marker) + src/main/resources/META-INF/services/ + ...HierarchyDefinitionService$HierarchyRuleProvider — SPI registration + src/test/java/.../compiler/ HierarchyRuleScriptParserTest.java — 5 parser tests HierarchyRuleClassGeneratorTest.java — 4 generator tests @@ -47,8 +51,9 @@ oap-server/analyzer/hierarchy/ | Component | Package / Name | |-----------|---------------| | Parser/Model/Generator | `org.apache.skywalking.oap.server.core.config.compiler` | -| Generated classes | `org.apache.skywalking.oap.server.core.config.compiler.rt.HierarchyRule_` | -| Package holder | `org.apache.skywalking.oap.server.core.config.compiler.rt.HierarchyRulePackageHolder` | +| Generated classes | `org.apache.skywalking.oap.server.core.config.compiler.hierarchy.rule.rt.HierarchyRule_` | +| Package holder | `org.apache.skywalking.oap.server.core.config.compiler.hierarchy.rule.rt.HierarchyRulePackageHolder` | +| SPI provider | `org.apache.skywalking.oap.server.core.config.compiler.CompiledHierarchyRuleProvider` | | Service type | `org.apache.skywalking.oap.server.core.query.type.Service` (in server-core) | `` is a global `AtomicInteger` counter. diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/CompiledHierarchyRuleProvider.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/CompiledHierarchyRuleProvider.java new file mode 100644 index 000000000000..00c9b589756e --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/CompiledHierarchyRuleProvider.java @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.core.config.compiler; + +import java.util.HashMap; +import java.util.Map; +import java.util.function.BiFunction; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.server.core.config.HierarchyDefinitionService; +import org.apache.skywalking.oap.server.core.query.type.Service; + +/** + * SPI implementation of {@link HierarchyDefinitionService.HierarchyRuleProvider} + * that compiles hierarchy matching rule expressions using ANTLR4 + Javassist. + * + *

Discovered at startup via {@code ServiceLoader} by + * {@link HierarchyDefinitionService}. For each rule expression + * (e.g., {@code "{ (u, l) -> u.name == l.name }"}): + *

    + *
  1. {@link HierarchyRuleClassGenerator#compile} parses the expression + * with ANTLR4 into an AST, then generates a Java class implementing + * {@code BiFunction} via Javassist.
    2. + *
  2. The generated class casts both arguments to {@link Service}, + * evaluates the expression body, and returns a {@code Boolean}.
    3. + *
+ * + *

The compiled matchers are returned to {@link HierarchyDefinitionService} + * and used at runtime by + * {@link org.apache.skywalking.oap.server.core.hierarchy.HierarchyService} + * to match service pairs. + */ +@Slf4j +public class CompiledHierarchyRuleProvider implements HierarchyDefinitionService.HierarchyRuleProvider { + + private final HierarchyRuleClassGenerator generator = new HierarchyRuleClassGenerator(); + + @Override + public Map> buildRules( + final Map ruleExpressions) { + final Map> rules = new HashMap<>(); + ruleExpressions.forEach((name, expression) -> { + try { + rules.put(name, generator.compile(name, expression)); + log.debug("Compiled hierarchy rule: {}", name); + } catch (Exception e) { + throw new IllegalStateException( + "Failed to compile hierarchy rule: " + name + + ", expression: " + expression, e); + } + }); + return rules; + } +} diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java index 754bffa27881..e6a460feb417 100644 --- a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGenerator.java @@ -24,19 +24,21 @@ import javassist.CtClass; import javassist.CtNewConstructor; import javassist.CtNewMethod; -import org.apache.skywalking.oap.server.core.config.compiler.rt.HierarchyRulePackageHolder; +import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.server.core.config.compiler.hierarchy.rule.rt.HierarchyRulePackageHolder; import org.apache.skywalking.oap.server.core.query.type.Service; /** * Generates {@link BiFunction BiFunction<Service, Service, Boolean>} implementation classes * from {@link HierarchyRuleModel} AST using Javassist bytecode generation. */ +@Slf4j public final class HierarchyRuleClassGenerator { private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(0); private static final String PACKAGE_PREFIX = - "org.apache.skywalking.oap.server.core.config.compiler.rt."; + "org.apache.skywalking.oap.server.core.config.compiler.hierarchy.rule.rt."; private final ClassPool classPool; @@ -68,6 +70,12 @@ public BiFunction compile( ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); final String applyBody = generateApplyMethod(model); + + if (log.isDebugEnabled()) { + log.debug("Hierarchy compile [{}] AST: {}", ruleName, model); + log.debug("Hierarchy compile [{}] apply():\n{}", ruleName, applyBody); + } + ctClass.addMethod(CtNewMethod.make(applyBody, ctClass)); final Class clazz = ctClass.toClass(HierarchyRulePackageHolder.class); diff --git a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/hierarchy/rule/rt/HierarchyRulePackageHolder.java similarity index 92% rename from oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java rename to oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/hierarchy/rule/rt/HierarchyRulePackageHolder.java index 31d74d245d67..367c1c541e65 100644 --- a/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/rt/HierarchyRulePackageHolder.java +++ b/oap-server/analyzer/hierarchy/src/main/java/org/apache/skywalking/oap/server/core/config/compiler/hierarchy/rule/rt/HierarchyRulePackageHolder.java @@ -15,7 +15,7 @@ * limitations under the License. */ -package org.apache.skywalking.oap.server.core.config.compiler.rt; +package org.apache.skywalking.oap.server.core.config.compiler.hierarchy.rule.rt; /** * Empty marker class used as the class loading anchor for Javassist diff --git a/oap-server/analyzer/hierarchy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.core.config.HierarchyDefinitionService$HierarchyRuleProvider b/oap-server/analyzer/hierarchy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.core.config.HierarchyDefinitionService$HierarchyRuleProvider new file mode 100644 index 000000000000..734565b7256d --- /dev/null +++ b/oap-server/analyzer/hierarchy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.core.config.HierarchyDefinitionService$HierarchyRuleProvider @@ -0,0 +1 @@ +org.apache.skywalking.oap.server.core.config.compiler.CompiledHierarchyRuleProvider diff --git a/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java b/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java index 1054bcb27bb0..816d12be418b 100644 --- a/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java +++ b/oap-server/analyzer/hierarchy/src/test/java/org/apache/skywalking/oap/server/core/config/compiler/HierarchyRuleClassGeneratorTest.java @@ -25,6 +25,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; class HierarchyRuleClassGeneratorTest { @@ -119,4 +120,39 @@ void compileLowerShortNameWithFqdn() throws Exception { upper.setShortName("no-port"); assertFalse(fn.apply(upper, lower)); } + + // ==================== Error handling tests ==================== + + @Test + void emptyExpressionThrows() { + // Demo error: Hierarchy rule parsing failed: 1:0 mismatched input '' + // expecting '{' + assertThrows(Exception.class, + () -> generator.compile("empty", "")); + } + + @Test + void missingClosureBracesThrows() { + // Demo error: Hierarchy rule parsing failed: 1:0 mismatched input 'u' + // expecting '{' + assertThrows(Exception.class, + () -> generator.compile("test", "u.name == l.name")); + } + + @Test + void missingParametersThrows() { + // Demo error: Hierarchy rule parsing failed: 1:2 mismatched input '}' + // expecting '(' + assertThrows(Exception.class, + () -> generator.compile("test", "{ }")); + } + + @Test + void invalidFieldAccessThrows() { + // Demo error: [source error] getNonExistent() not found in Service + // (Javassist cannot find the getter for a non-existent field) + assertThrows(Exception.class, + () -> generator.compile("test", + "{ (u, l) -> u.nonExistent == l.nonExistent }")); + } } diff --git a/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 b/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 index b381aa157369..8c90cc344d8a 100644 --- a/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 +++ b/oap-server/analyzer/log-analyzer/src/main/antlr4/org/apache/skywalking/lal/rt/grammar/LALParser.g4 @@ -404,6 +404,7 @@ valueAccess valueAccessPrimary : PARSED # valueParsed | LOG # valueLog + | PROCESS_REGISTRY # valueProcessRegistry | IDENTIFIER # valueIdentifier | STRING # valueString | NUMBER # valueNumber @@ -420,7 +421,12 @@ valueAccessSegment ; functionInvocation - : IDENTIFIER L_PAREN functionArgList? R_PAREN + : functionName L_PAREN functionArgList? R_PAREN + ; + +functionName + : IDENTIFIER + | TAG ; functionArgList diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java index 092d9f45f126..a5cdcab939ea 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java @@ -26,6 +26,7 @@ import javassist.CtField; import javassist.CtNewConstructor; import javassist.CtNewMethod; +import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.log.analyzer.compiler.rt.LalExpressionPackageHolder; import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression; @@ -37,6 +38,7 @@ * Consumer callbacks are pre-compiled as separate classes and * stored as fields on the main class. */ +@Slf4j public final class LALClassGenerator { private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(0); @@ -106,6 +108,12 @@ public LalExpression compileFromModel(final LALScriptModel model) throws Excepti // Phase 4: Generate execute method referencing consumer fields final int[] counter = {0}; final String executeBody = generateExecuteMethod(model, counter); + + if (log.isDebugEnabled()) { + log.debug("LAL compile AST: {}", model); + log.debug("LAL compile execute():\n{}", executeBody); + } + ctClass.addMethod(CtNewMethod.make(executeBody, ctClass)); final Class clazz = ctClass.toClass(LalExpressionPackageHolder.class); @@ -221,16 +229,36 @@ private void generateExtractorStatementsFlat( } else if (stmt instanceof LALScriptModel.TagAssignment) { final LALScriptModel.TagAssignment tag = (LALScriptModel.TagAssignment) stmt; - if (tag.getTags().size() == 1) { - final Map.Entry entry = - tag.getTags().entrySet().iterator().next(); - sb.append(" _t.tag(\"") - .append(escapeJava(entry.getKey())).append("\", "); - generateCastedValueAccess(sb, entry.getValue().getValue(), - entry.getValue().getCastType()); - sb.append(");\n"); - } + generateTagAssignment(sb, tag); + } + } + } + + private void generateTagAssignment(final StringBuilder sb, + final LALScriptModel.TagAssignment tag) { + final Map tags = tag.getTags(); + if (tags.isEmpty()) { + return; + } + if (tags.size() == 1) { + final Map.Entry entry = + tags.entrySet().iterator().next(); + sb.append(" _t.tag(java.util.Collections.singletonMap(\"") + .append(escapeJava(entry.getKey())).append("\", "); + generateCastedValueAccess(sb, entry.getValue().getValue(), + entry.getValue().getCastType()); + sb.append("));\n"); + } else { + sb.append(" { java.util.Map _tagMap = new java.util.LinkedHashMap();\n"); + for (final Map.Entry entry + : tags.entrySet()) { + sb.append(" _tagMap.put(\"") + .append(escapeJava(entry.getKey())).append("\", "); + generateCastedValueAccess(sb, entry.getValue().getValue(), + entry.getValue().getCastType()); + sb.append(");\n"); } + sb.append(" _t.tag(_tagMap); }\n"); } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java index 128f32557362..9bd9d241c1b5 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java @@ -538,7 +538,15 @@ private static Condition makeComparison( return new ComparisonCondition(left, leftCast, op, visitConditionExprAsValue(rightCtx)); } - // For function calls and other forms, wrap as expression condition + if (leftCtx instanceof LALParser.CondFunctionCallContext) { + final LALParser.FunctionInvocationContext fi = + ((LALParser.CondFunctionCallContext) leftCtx).functionInvocation(); + final ValueAccess left = new ValueAccess( + List.of(fi.getText()), false, false, List.of()); + return new ComparisonCondition(left, null, op, + visitConditionExprAsValue(rightCtx)); + } + // For other forms, wrap as expression condition return new ExprCondition( new ValueAccess(List.of(leftCtx.getText()), false, false, List.of()), null); } @@ -592,6 +600,8 @@ private static ValueAccess visitValueAccess(final LALParser.ValueAccessContext c } else if (primary instanceof LALParser.ValueLogContext) { logRef = true; segments.add("log"); + } else if (primary instanceof LALParser.ValueProcessRegistryContext) { + segments.add("ProcessRegistry"); } else if (primary instanceof LALParser.ValueIdentifierContext) { segments.add(((LALParser.ValueIdentifierContext) primary).IDENTIFIER().getText()); } else if (primary instanceof LALParser.ValueStringContext) { @@ -622,15 +632,15 @@ private static ValueAccess visitValueAccess(final LALParser.ValueAccessContext c } else if (seg instanceof LALParser.SegmentMethodContext) { final LALParser.FunctionInvocationContext fi = ((LALParser.SegmentMethodContext) seg).functionInvocation(); - segments.add(fi.IDENTIFIER().getText() + "()"); + segments.add(fi.functionName().getText() + "()"); chain.add(new LALScriptModel.MethodSegment( - fi.IDENTIFIER().getText(), List.of(), false)); + fi.functionName().getText(), List.of(), false)); } else if (seg instanceof LALParser.SegmentSafeMethodContext) { final LALParser.FunctionInvocationContext fi = ((LALParser.SegmentSafeMethodContext) seg).functionInvocation(); - segments.add(fi.IDENTIFIER().getText() + "()"); + segments.add(fi.functionName().getText() + "()"); chain.add(new LALScriptModel.MethodSegment( - fi.IDENTIFIER().getText(), List.of(), true)); + fi.functionName().getText(), List.of(), true)); } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java index ee2beaa47403..41404fbdb083 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java @@ -33,9 +33,22 @@ import org.apache.skywalking.oap.server.core.source.Log; /** - * Same-FQCN replacement for upstream Binding. - * Pure Java implementation that does not extend groovy.lang.Binding. - * Uses a simple HashMap for property storage instead of Groovy's binding mechanism. + * Mutable property storage for a single LAL script execution cycle. + * + *

A new Binding is created for each incoming log in + * {@link org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogFilterListener#parse}. + * It carries all per-log state through the compiled LAL pipeline: + *

    + *
  • {@code log} — the incoming {@code LogData.Builder}
    • + *
  • {@code parsed} — structured data extracted by json/text/yaml parsers
    • + *
  • {@code save}/{@code abort} — control flags set by extractor/sink logic
    • + *
  • {@code metrics_container} — optional list for LAL-extracted metrics (log-MAL)
    • + *
  • {@code log_container} — optional container for the built {@code Log} source object
    • + *
+ * + *

The Binding is injected into {@link org.apache.skywalking.oap.log.analyzer.dsl.spec.AbstractSpec} + * via a ThreadLocal ({@code BINDING}), so all Spec methods ({@code FilterSpec}, {@code ExtractorSpec}, + * {@code SinkSpec}) can access the current log data without explicit parameter passing. */ public class Binding { public static final String KEY_LOG = "log"; diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java index b3f4e4c977d2..a7d37b7b20e7 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java @@ -17,35 +17,45 @@ package org.apache.skywalking.oap.log.analyzer.dsl; -import java.io.BufferedReader; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.nio.charset.StandardCharsets; -import java.security.MessageDigest; -import java.security.NoSuchAlgorithmException; -import java.util.HashMap; -import java.util.Map; -import java.util.concurrent.atomic.AtomicInteger; import lombok.AccessLevel; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.log.analyzer.compiler.LALClassGenerator; import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; import org.apache.skywalking.oap.server.library.module.ModuleManager; import org.apache.skywalking.oap.server.library.module.ModuleStartException; /** - * Same-FQCN replacement for upstream LAL DSL. - * Loads pre-compiled {@link LalExpression} classes from lal-expressions.txt manifest - * (keyed by SHA-256 hash) instead of Groovy {@code GroovyShell} runtime compilation. + * DSL compiles a LAL (Log Analysis Language) expression string into a + * {@link LalExpression} object and wraps it with runtime state management. + * + *

One DSL instance is created per LAL rule entry defined in a {@code .yaml} + * config file under {@code lal/}. Instances are compiled once at startup and + * reused for every incoming log. + * + *

Compilation ({@link #of}): + * 

+ *   LAL DSL string (e.g., "filter { json {} extractor { service ... } sink {} }")
+ *     → LALClassGenerator.compile(dsl)
+ *       → ANTLR4 parse → AST → Javassist bytecode → LalExpression class
+ *     → new FilterSpec(moduleManager, config)
+ *     → DSL(expression, filterSpec)
+ *
+ * + *

Runtime (per-log execution by {@link org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogFilterListener}): + *

    + *
  1. {@link #bind(Binding)} — sets the current log data into the {@link FilterSpec} + * via a ThreadLocal, making it available to all Spec methods.
    2. + *
  2. {@link #evaluate()} — invokes the compiled {@link LalExpression#execute}, + * which calls FilterSpec methods (json/text/yaml, extractor, sink) in the + * order defined by the LAL script.
    3. + *
*/ @Slf4j @RequiredArgsConstructor(access = AccessLevel.PRIVATE) public class DSL { - private static final String MANIFEST_PATH = "META-INF/lal-expressions.txt"; - private static volatile Map EXPRESSION_MAP; - private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + private static final LALClassGenerator GENERATOR = new LALClassGenerator(); private final LalExpression expression; private final FilterSpec filterSpec; @@ -54,28 +64,13 @@ public class DSL { public static DSL of(final ModuleManager moduleManager, final LogAnalyzerModuleConfig config, final String dsl) throws ModuleStartException { - final Map exprMap = loadManifest(); - final String dslHash = sha256(dsl); - final String className = exprMap.get(dslHash); - if (className == null) { - throw new ModuleStartException( - "Pre-compiled LAL expression not found for DSL hash: " + dslHash - + ". Available: " + exprMap.size() + " expressions."); - } - try { - final Class exprClass = Class.forName(className); - final LalExpression expression = (LalExpression) exprClass.getDeclaredConstructor().newInstance(); + final LalExpression expression = GENERATOR.compile(dsl); final FilterSpec filterSpec = new FilterSpec(moduleManager, config); - final int count = LOADED_COUNT.incrementAndGet(); - log.debug("Loaded pre-compiled LAL expression [{}/{}]: {}", count, exprMap.size(), className); return new DSL(expression, filterSpec); - } catch (ClassNotFoundException e) { + } catch (Exception e) { throw new ModuleStartException( - "Pre-compiled LAL expression class not found: " + className, e); - } catch (ReflectiveOperationException e) { - throw new ModuleStartException( - "Pre-compiled LAL expression instantiation failed: " + className, e); + "Failed to compile LAL expression: " + dsl, e); } } @@ -87,56 +82,4 @@ public void bind(final Binding binding) { public void evaluate() { expression.execute(filterSpec, binding); } - - private static Map loadManifest() { - if (EXPRESSION_MAP != null) { - return EXPRESSION_MAP; - } - synchronized (DSL.class) { - if (EXPRESSION_MAP != null) { - return EXPRESSION_MAP; - } - final Map map = new HashMap<>(); - try (InputStream is = DSL.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { - if (is == null) { - log.warn("LAL expression manifest not found: {}", MANIFEST_PATH); - EXPRESSION_MAP = map; - return map; - } - try (BufferedReader reader = new BufferedReader( - new InputStreamReader(is, StandardCharsets.UTF_8))) { - String line; - while ((line = reader.readLine()) != null) { - line = line.trim(); - if (line.isEmpty()) { - continue; - } - final String[] parts = line.split("=", 2); - if (parts.length == 2) { - map.put(parts[0], parts[1]); - } - } - } - } catch (IOException e) { - throw new IllegalStateException("Failed to load LAL expression manifest", e); - } - log.info("Loaded {} pre-compiled LAL expressions from manifest", map.size()); - EXPRESSION_MAP = map; - return map; - } - } - - static String sha256(final String input) { - try { - final MessageDigest digest = MessageDigest.getInstance("SHA-256"); - final byte[] hash = digest.digest(input.getBytes(StandardCharsets.UTF_8)); - final StringBuilder hex = new StringBuilder(); - for (final byte b : hash) { - hex.append(String.format("%02x", b)); - } - return hex.toString(); - } catch (NoSuchAlgorithmException e) { - throw new IllegalStateException("SHA-256 not available", e); - } - } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java index f96b02f485a2..f40c7e95c01e 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java @@ -21,8 +21,13 @@ import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; /** - * Pure Java replacement for Groovy-based LAL DelegatingScript. - * Each transpiled LAL expression implements this interface. + * Functional interface implemented by each compiled LAL class. + * + *

Generated at startup by + * {@link org.apache.skywalking.oap.log.analyzer.compiler.LALClassGenerator} + * via ANTLR4 parsing and Javassist bytecode generation. + * The generated {@code execute} method calls {@link FilterSpec} methods + * (json/text/yaml, extractor, sink) in the order defined by the LAL script. */ @FunctionalInterface public interface LalExpression { diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java index c9e3338df69a..34ac9a7db129 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java @@ -47,6 +47,27 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; +/** + * The top-level runtime API that compiled LAL expressions invoke. + * + *

A compiled {@link org.apache.skywalking.oap.log.analyzer.dsl.LalExpression} + * calls methods on this class in the order defined by the LAL script: + *

    + *
  1. Parser: {@code json()}, {@code text()}, or {@code yaml()} — parses the log body + * into structured data stored in {@link Binding#parsed()}.
    2. + *
  2. Extractor: {@code extractor(Consumer)} — extracts service name, layer, tags, + * metrics, slow SQL, sampled traces, etc.
    3. + *
  3. Sink: {@code sink()} or {@code sink(Consumer)} — materializes the log into + * storage via {@link LogSinkListenerFactory} instances (RecordSinkListener, + * TrafficSinkListener), unless the log has been dropped or aborted.
    4. + *
+ * + *

All methods read the current log data from the ThreadLocal {@code BINDING} + * (inherited from {@link AbstractSpec}), which is set by + * {@link org.apache.skywalking.oap.log.analyzer.dsl.DSL#bind(Binding)} before each execution. + * Every method checks {@code shouldAbort()} first and short-circuits if a previous + * step aborted the pipeline. + */ public class FilterSpec extends AbstractSpec { private static final Logger LOGGER = LoggerFactory.getLogger(FilterSpec.class); diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java index 73909b1813e2..b9e5b57d70c3 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java @@ -33,7 +33,24 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager; /** - * Analyze the collected log data, is the entry point for log analysis. + * Entry point for log analysis. Created per-request by the log receiver. + * + *

Runtime execution ({@link #doAnalysis}): + *

    + *
  1. Validates the incoming log (service name must be non-empty, layer must be valid).
    2. + *
  2. Calls {@code createAnalysisListeners(layer)} — asks all registered + * {@link org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogAnalysisListenerFactory} + * instances to create listeners for the log's layer. For LAL, this is + * {@link org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogFilterListener.Factory}, + * which returns a listener wrapping all compiled {@link org.apache.skywalking.oap.log.analyzer.dsl.DSL} + * instances for that layer.
    3. + *
  3. {@code notifyAnalysisListener(builder, extraLog)} — calls + * {@link org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogAnalysisListener#parse} + * on each listener, which binds the log data to the compiled LAL scripts.
    4. + *
  4. {@code notifyAnalysisListenerToBuild()} — calls + * {@link org.apache.skywalking.oap.log.analyzer.provider.log.listener.LogAnalysisListener#build} + * on each listener, which evaluates the compiled LAL scripts (extractors, sinks).
    5. + *
*/ @Slf4j @RequiredArgsConstructor diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java index 8b9f0b1fb80c..75af6b10127e 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java @@ -38,6 +38,29 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager; import org.apache.skywalking.oap.server.library.module.ModuleStartException; +/** + * Runtime listener that executes compiled LAL rules against incoming log data. + * + *

Each instance wraps a collection of {@link DSL} objects — one per LAL rule + * defined for a specific {@link Layer}. Created per-log by {@link Factory#create(Layer)}. + * + *

Two-phase execution (called by {@link org.apache.skywalking.oap.log.analyzer.provider.log.LogAnalyzer}): + *

    + *
  1. {@link #parse} — creates a fresh {@link Binding} with the current log data + * and binds it to every DSL instance (sets the ThreadLocal in each Spec).
    2. + *
  2. {@link #build} — calls {@link DSL#evaluate()} on every DSL instance, + * which invokes the compiled {@link org.apache.skywalking.oap.log.analyzer.dsl.LalExpression} + * to run the filter/extractor/sink pipeline.
    3. + *
+ * + *

The inner {@link Factory} is created once at startup by + * {@link org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleProvider#start()}. + * It loads all {@code .yaml} LAL config files, compiles each rule's DSL string + * into a {@link DSL} instance via + * {@link DSL#of(org.apache.skywalking.oap.server.library.module.ModuleManager, + * org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig, String)}, + * and organizes them by {@link Layer}. + */ @Slf4j @RequiredArgsConstructor public class LogFilterListener implements LogAnalysisListener { diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java index 7bde461a4f76..11cee26a833c 100644 --- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java @@ -23,6 +23,7 @@ import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; class LALClassGeneratorTest { @@ -94,4 +95,36 @@ void generateSourceReturnsJavaCode() { org.junit.jupiter.api.Assertions.assertTrue( source.contains("filterSpec.sink()")); } + + // ==================== Error handling tests ==================== + + @Test + void emptyScriptThrows() { + // Demo error: LAL script parsing failed: 1:0 mismatched input '' + // expecting 'filter' + assertThrows(Exception.class, () -> generator.compile("")); + } + + @Test + void missingFilterKeywordThrows() { + // Demo error: LAL script parsing failed: 1:0 extraneous input 'json' + // expecting 'filter' + assertThrows(Exception.class, () -> generator.compile("json {}")); + } + + @Test + void unclosedBraceThrows() { + // Demo error: LAL script parsing failed: 1:15 mismatched input '' + // expecting '}' + assertThrows(Exception.class, + () -> generator.compile("filter { json {")); + } + + @Test + void invalidStatementInFilterThrows() { + // Demo error: LAL script parsing failed: 1:9 extraneous input 'invalid' + // expecting {'text', 'json', 'yaml', 'extractor', 'sink', 'abort', 'if', '}'} + assertThrows(Exception.class, + () -> generator.compile("filter { invalid {} }")); + } } diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java index a8f19d382e41..8a8178f6c680 100644 --- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLV2Test.java @@ -17,44 +17,33 @@ package org.apache.skywalking.oap.log.analyzer.dsl; -import org.apache.skywalking.oap.server.library.module.ModuleStartException; +import org.apache.skywalking.oap.log.analyzer.compiler.LALClassGenerator; import org.junit.jupiter.api.Test; -import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertThrows; class DSLV2Test { @Test - void ofThrowsWhenManifestMissing() { - // No META-INF/lal-expressions.txt on test classpath - assertThrows(ModuleStartException.class, - () -> DSL.of(null, null, "filter { json {} sink {} }")); + void compileSimpleFilterExpression() throws Exception { + final LALClassGenerator generator = new LALClassGenerator(); + final LalExpression expr = generator.compile("filter { json {} sink {} }"); + assertNotNull(expr); } @Test - void sha256Deterministic() { - final String input = "filter { json {} sink {} }"; - final String hash1 = DSL.sha256(input); - final String hash2 = DSL.sha256(input); - assertNotNull(hash1); - assertEquals(64, hash1.length()); - assertEquals(hash1, hash2); + void compileFilterWithExtractor() throws Exception { + final LALClassGenerator generator = new LALClassGenerator(); + final LalExpression expr = generator.compile( + "filter { json {} extractor { service parsed.service as String } sink {} }"); + assertNotNull(expr); } @Test - void sha256DifferentInputsDifferentHashes() { - final String hash1 = DSL.sha256("filter { json {} sink {} }"); - final String hash2 = DSL.sha256("filter { text {} sink {} }"); - assertNotNull(hash1); - assertNotNull(hash2); - assertNotEquals(hash1, hash2); - } - - private static void assertNotEquals(final String a, final String b) { - if (a.equals(b)) { - throw new AssertionError("Expected different values but got: " + a); - } + void compileThrowsOnInvalidExpression() { + final LALClassGenerator generator = new LALClassGenerator(); + assertThrows(Exception.class, + () -> generator.compile("??? invalid !!!")); } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 b/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 index 59af9b14d03e..0306ec4a2be1 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 +++ b/oap-server/analyzer/meter-analyzer/src/main/antlr4/org/apache/skywalking/mal/rt/grammar/MALParser.g4 @@ -37,6 +37,11 @@ expression : additiveExpression EOF ; +// A standalone filter closure: { tags -> tags.job_name == 'value' } +filterExpression + : closureExpression EOF + ; + // ==================== Arithmetic ==================== additiveExpression @@ -123,8 +128,10 @@ closureParams ; closureBody - : closureStatement+ - | L_BRACE closureStatement+ R_BRACE // optional extra braces: { tags -> { ... } } + : closureCondition // bare condition: { tags -> tags.x == 'v' } + | L_BRACE closureCondition R_BRACE // braced condition: { tags -> { tags.x == 'v' } } + | closureStatement+ + | L_BRACE closureStatement+ R_BRACE // optional extra braces: { tags -> { ... } } ; closureStatement diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java index 0d698a9b6b98..9b87b7d92aa3 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java @@ -69,6 +69,29 @@ /** * Analyzer analyses DSL expression with input samples, then to generate meter-system metrics. + * + *

One Analyzer is created per {@code metricsRules} entry in a MAL config YAML file. + * + *

Initialization ({@link #build}): + *

    + *
  1. Compiles the MAL expression string into a + * {@link org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression MalExpression} + * via ANTLR4 + Javassist.
    2. + *
  2. Extracts compile-time {@link ExpressionMetadata} from the AST (sample names, scope type, + * aggregation labels, downsampling, histogram/percentile info).
    3. + *
  3. Registers the metric in {@link MeterSystem} (generates storage class via Javassist).
    4. + *
+ * + *

Runtime ({@link #analyse}): + *

    + *
  1. Receives the full {@code sampleFamilies} map (all metrics from one scrape).
    2. + *
  2. Selects only the entries matching {@code this.samples} (e.g., ["node_cpu_seconds_total"]). + * This is an O(n) lookup where n is the number of input metric names in the expression + * (typically 1-2), not the size of the full map.
    3. + *
  3. Applies the optional tag filter (e.g., {@code job_name == 'vm-monitoring'}).
    4. + *
  4. Executes the compiled MAL expression on the filtered input.
    5. + *
  5. Sends computed metric values to MeterSystem for storage.
    6. + *
*/ @Slf4j @RequiredArgsConstructor(access = AccessLevel.PRIVATE) @@ -112,11 +135,17 @@ public static Analyzer build(final String metricName, private int[] percentiles; /** - * analyse intends to parse expression with input samples to meter-system metrics. + * Analyse the full sample family map and produce meter-system metrics. + * + *

The {@code sampleFamilies} map contains ALL metrics from one scrape batch. + * This method first selects only the entries matching {@code this.samples} + * (the input metric names extracted from the MAL expression AST at compile time), + * then applies the optional filter and runs the expression on the selected subset. * - * @param sampleFamilies input samples. + * @param sampleFamilies all sample families from one scrape, keyed by metric name. */ public void analyse(final ImmutableMap sampleFamilies) { + // Select only the metric families this expression references (typically 1-2 keys). Map input = samples.stream() .map(s -> Tuple.of(s, sampleFamilies.get(s))) .filter(t -> t._2 != null) @@ -232,6 +261,13 @@ private enum MetricType { private final String literal; } + /** + * Initializes runtime state from compile-time metadata. + * + *

{@code ctx.getSamples()} provides the Prometheus metric names this expression references + * (e.g., ["node_cpu_seconds_total"]). These are used at runtime to select relevant entries + * from the full sample family map, avoiding unnecessary expression evaluation. + */ private void init() { this.samples = ctx.getSamples(); if (ctx.isHistogram()) { diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java index 5c89f42b5edf..ffcfa159b6b3 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java @@ -38,6 +38,30 @@ /** * MetricConvert converts {@link SampleFamily} collection to meter-system metrics, then store them to backend storage. + * + *

One MetricConvert instance is created per MAL config YAML file (e.g., {@code vm.yaml}). + * It holds a list of {@link Analyzer}s, one per {@code metricsRules} entry in the YAML. + * + *

Construction (at startup): + * 

+ *   YAML file (e.g., vm.yaml)
+ *     metricPrefix: meter_vm
+ *     expSuffix:    service(['host'], Layer.OS_LINUX)
+ *     filter:       { tags -> tags.job_name == 'vm-monitoring' }
+ *     metricsRules:
+ *       - name: cpu_total_percentage
+ *         exp:  (node_cpu_seconds_total * 100).sum(['host']).rate('PT1M')
+ *
+ *   MetricConvert(rule, meterSystem)
+ *     for each rule:
+ *       metricName = metricPrefix + "_" + name    → "meter_vm_cpu_total_percentage"
+ *       finalExp   = (exp).expSuffix              → "(...).service(['host'], Layer.OS_LINUX)"
+ *       → Analyzer.build(metricName, filter, finalExp, meterSystem)
+ *
+ * + *

Runtime ({@link #toMeter}): receives the full {@code sampleFamilies} map (all metrics + * from one scrape) and broadcasts it to every Analyzer. Each Analyzer self-filters to only + * the input metrics it needs (via {@code this.samples} from compile-time metadata). */ @Slf4j public class MetricConvert { @@ -95,9 +119,14 @@ private String formatExp(final String expPrefix, String expSuffix, String exp) { } /** - * toMeter transforms {@link SampleFamily} collection to meter-system metrics. + * Broadcasts the full sample family map to every Analyzer in this config file. + * + *

The map contains ALL metrics from a single scrape batch keyed by Prometheus metric name + * (e.g., "node_cpu_seconds_total", "node_memory_MemTotal_bytes", ...). + * Each Analyzer selects only the entries it needs via O(1) HashMap lookups on + * {@code this.samples} (derived from compile-time AST metadata). * - * @param sampleFamilies {@link SampleFamily} collection. + * @param sampleFamilies all sample families from one scrape, keyed by metric name. */ public void toMeter(final ImmutableMap sampleFamilies) { Preconditions.checkNotNull(sampleFamilies); diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java index 0d42dd12aaf8..e2f10c796695 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGenerator.java @@ -18,7 +18,6 @@ package org.apache.skywalking.oap.meter.analyzer.compiler; import java.util.ArrayList; -import java.util.HashSet; import java.util.LinkedHashSet; import java.util.List; import java.util.Set; @@ -27,10 +26,12 @@ import javassist.CtClass; import javassist.CtNewConstructor; import javassist.CtNewMethod; +import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.meter.analyzer.compiler.rt.MalExpressionPackageHolder; import org.apache.skywalking.oap.meter.analyzer.dsl.DownsamplingType; import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionMetadata; import org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression; +import org.apache.skywalking.oap.meter.analyzer.dsl.MalFilter; import org.apache.skywalking.oap.server.core.analysis.meter.ScopeType; /** @@ -42,6 +43,7 @@ * SampleFamily run(Map<String, SampleFamily> samples) * */ +@Slf4j public final class MALClassGenerator { private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(0); @@ -93,6 +95,73 @@ public MalExpression compile(final String metricName, return compileFromModel(metricName, ast); } + /** + * Compiles a MAL filter closure into a {@link MalFilter} implementation. + * + * @param filterExpression e.g. {@code "{ tags -> tags.job_name == 'mysql-monitoring' }"} + * @return a MalFilter instance + */ + @SuppressWarnings("unchecked") + public MalFilter compileFilter(final String filterExpression) throws Exception { + final MALExpressionModel.ClosureArgument closure = + MALScriptParser.parseFilter(filterExpression); + + final String className = PACKAGE_PREFIX + "MalFilter_" + + CLASS_COUNTER.getAndIncrement(); + + final CtClass ctClass = classPool.makeClass(className); + ctClass.addInterface(classPool.get( + "org.apache.skywalking.oap.meter.analyzer.dsl.MalFilter")); + + ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); + + final List params = closure.getParams(); + final String paramName = params.isEmpty() ? "it" : params.get(0); + + final StringBuilder sb = new StringBuilder(); + sb.append("public boolean test(java.util.Map ").append(paramName) + .append(") {\n"); + + final List body = closure.getBody(); + if (body.size() == 1 + && body.get(0) instanceof MALExpressionModel.ClosureExprStatement) { + // Single expression → evaluate as condition and return boolean + final MALExpressionModel.ClosureExpr expr = + ((MALExpressionModel.ClosureExprStatement) body.get(0)).getExpr(); + if (expr instanceof MALExpressionModel.ClosureCondition) { + sb.append(" return "); + generateClosureCondition( + sb, (MALExpressionModel.ClosureCondition) expr, paramName); + sb.append(";\n"); + } else { + // Truthy evaluation of the expression + sb.append(" Object _v = "); + generateClosureExpr(sb, expr, paramName); + sb.append(";\n"); + sb.append(" return _v != null && !Boolean.FALSE.equals(_v);\n"); + } + } else { + // Multi-statement body — generate statements, last expression is the return + for (final MALExpressionModel.ClosureStatement stmt : body) { + generateClosureStatement(sb, stmt, paramName); + } + sb.append(" return false;\n"); + } + sb.append("}\n"); + + final String filterBody = sb.toString(); + if (log.isDebugEnabled()) { + log.debug("MAL compileFilter AST: {}", closure); + log.debug("MAL compileFilter test():\n{}", filterBody); + } + + ctClass.addMethod(CtNewMethod.make(filterBody, ctClass)); + + final Class clazz = ctClass.toClass(MalExpressionPackageHolder.class); + ctClass.detach(); + return (MalFilter) clazz.getDeclaredConstructor().newInstance(); + } + /** * Compiles from a pre-parsed AST model. */ @@ -128,10 +197,16 @@ public MalExpression compileFromModel(final String metricName, ctClass.addConstructor(CtNewConstructor.defaultConstructor(ctClass)); final String runBody = generateRunMethod(ast); - ctClass.addMethod(CtNewMethod.make(runBody, ctClass)); - final ExpressionMetadata metadata = extractMetadata(ast); final String metadataBody = generateMetadataMethod(metadata); + + if (log.isDebugEnabled()) { + log.debug("MAL compile [{}] AST: {}", metricName, ast); + log.debug("MAL compile [{}] run():\n{}", metricName, runBody); + log.debug("MAL compile [{}] metadata():\n{}", metricName, metadataBody); + } + + ctClass.addMethod(CtNewMethod.make(runBody, ctClass)); ctClass.addMethod(CtNewMethod.make(metadataBody, ctClass)); final Class clazz = ctClass.toClass(MalExpressionPackageHolder.class); @@ -599,32 +674,49 @@ private void generateClosureMethodChain( ((MALExpressionModel.ClosureIndexAccess) segs.get(0)).getIndex(), paramName); sb.append(")"); } else { - // General chain - sb.append(chain.getTarget()); + // General chain: build in a local buffer to support safe navigation + final StringBuilder local = new StringBuilder(); + local.append(chain.getTarget()); for (final MALExpressionModel.ClosureChainSegment seg : segs) { if (seg instanceof MALExpressionModel.ClosureFieldAccess) { - sb.append(".get(\"") + local.append(".get(\"") .append(escapeJava( ((MALExpressionModel.ClosureFieldAccess) seg).getName())) .append("\")"); } else if (seg instanceof MALExpressionModel.ClosureIndexAccess) { - sb.append(".get("); - generateClosureExpr(sb, + local.append(".get("); + generateClosureExpr(local, ((MALExpressionModel.ClosureIndexAccess) seg).getIndex(), paramName); - sb.append(")"); + local.append(")"); } else if (seg instanceof MALExpressionModel.ClosureMethodCallSeg) { final MALExpressionModel.ClosureMethodCallSeg mc = (MALExpressionModel.ClosureMethodCallSeg) seg; - sb.append('.').append(mc.getName()).append('('); - for (int i = 0; i < mc.getArguments().size(); i++) { - if (i > 0) { - sb.append(", "); + if (mc.isSafeNav()) { + final String prior = local.toString(); + local.setLength(0); + local.append("(").append(prior).append(" == null ? null : ") + .append("((String) ").append(prior).append(").") + .append(mc.getName()).append('('); + for (int i = 0; i < mc.getArguments().size(); i++) { + if (i > 0) { + local.append(", "); + } + generateClosureExpr(local, mc.getArguments().get(i), paramName); + } + local.append("))"); + } else { + local.append('.').append(mc.getName()).append('('); + for (int i = 0; i < mc.getArguments().size(); i++) { + if (i > 0) { + local.append(", "); + } + generateClosureExpr(local, mc.getArguments().get(i), paramName); } - generateClosureExpr(sb, mc.getArguments().get(i), paramName); + local.append(')'); } - sb.append(')'); } } + sb.append(local); } } @@ -693,7 +785,9 @@ private void generateClosureCondition(final StringBuilder sb, private static void collectSampleNames(final MALExpressionModel.Expr expr, final Set names) { if (expr instanceof MALExpressionModel.MetricExpr) { - names.add(((MALExpressionModel.MetricExpr) expr).getMetricName()); + final MALExpressionModel.MetricExpr me = (MALExpressionModel.MetricExpr) expr; + names.add(me.getMetricName()); + collectSampleNamesFromChain(me.getMethodChain(), names); } else if (expr instanceof MALExpressionModel.BinaryExpr) { collectSampleNames(((MALExpressionModel.BinaryExpr) expr).getLeft(), names); collectSampleNames(((MALExpressionModel.BinaryExpr) expr).getRight(), names); @@ -701,8 +795,10 @@ private static void collectSampleNames(final MALExpressionModel.Expr expr, collectSampleNames( ((MALExpressionModel.UnaryNegExpr) expr).getOperand(), names); } else if (expr instanceof MALExpressionModel.ParenChainExpr) { - collectSampleNames( - ((MALExpressionModel.ParenChainExpr) expr).getInner(), names); + final MALExpressionModel.ParenChainExpr pce = + (MALExpressionModel.ParenChainExpr) expr; + collectSampleNames(pce.getInner(), names); + collectSampleNamesFromChain(pce.getMethodChain(), names); } else if (expr instanceof MALExpressionModel.FunctionCallExpr) { for (final MALExpressionModel.Argument arg : ((MALExpressionModel.FunctionCallExpr) expr).getArguments()) { @@ -714,11 +810,27 @@ private static void collectSampleNames(final MALExpressionModel.Expr expr, } } + private static void collectSampleNamesFromChain( + final List chain, + final Set names) { + for (final MALExpressionModel.MethodCall mc : chain) { + if ("downsampling".equals(mc.getName())) { + continue; + } + for (final MALExpressionModel.Argument arg : mc.getArguments()) { + if (arg instanceof MALExpressionModel.ExprArgument) { + collectSampleNames( + ((MALExpressionModel.ExprArgument) arg).getExpr(), names); + } + } + } + } + /** * Extracts compile-time metadata from the AST by walking all method chains. */ static ExpressionMetadata extractMetadata(final MALExpressionModel.Expr ast) { - final Set sampleNames = new HashSet<>(); + final Set sampleNames = new LinkedHashSet<>(); collectSampleNames(ast, sampleNames); ScopeType scopeType = null; @@ -786,11 +898,21 @@ static ExpressionMetadata extractMetadata(final MALExpressionModel.Expr ast) { } break; case "downsampling": - if (!mc.getArguments().isEmpty() - && mc.getArguments().get(0) instanceof MALExpressionModel.EnumRefArgument) { - final String val = - ((MALExpressionModel.EnumRefArgument) mc.getArguments().get(0)).getEnumValue(); - downsampling = DownsamplingType.valueOf(val); + if (!mc.getArguments().isEmpty()) { + final MALExpressionModel.Argument dsArg = mc.getArguments().get(0); + if (dsArg instanceof MALExpressionModel.EnumRefArgument) { + final String val = + ((MALExpressionModel.EnumRefArgument) dsArg).getEnumValue(); + downsampling = DownsamplingType.valueOf(val); + } else if (dsArg instanceof MALExpressionModel.ExprArgument) { + final MALExpressionModel.Expr dsExpr = + ((MALExpressionModel.ExprArgument) dsArg).getExpr(); + if (dsExpr instanceof MALExpressionModel.MetricExpr) { + final String val = + ((MALExpressionModel.MetricExpr) dsExpr).getMetricName(); + downsampling = DownsamplingType.valueOf(val); + } + } } break; default: @@ -968,4 +1090,44 @@ public String generateSource(final String expression) { final MALExpressionModel.Expr ast = MALScriptParser.parse(expression); return generateRunMethod(ast); } + + /** + * Generates the Java source body of the filter test method for debugging/testing. + */ + public String generateFilterSource(final String filterExpression) { + final MALExpressionModel.ClosureArgument closure = + MALScriptParser.parseFilter(filterExpression); + + final List params = closure.getParams(); + final String paramName = params.isEmpty() ? "it" : params.get(0); + + final StringBuilder sb = new StringBuilder(); + sb.append("public boolean test(java.util.Map ").append(paramName) + .append(") {\n"); + + final List body = closure.getBody(); + if (body.size() == 1 + && body.get(0) instanceof MALExpressionModel.ClosureExprStatement) { + final MALExpressionModel.ClosureExpr expr = + ((MALExpressionModel.ClosureExprStatement) body.get(0)).getExpr(); + if (expr instanceof MALExpressionModel.ClosureCondition) { + sb.append(" return "); + generateClosureCondition( + sb, (MALExpressionModel.ClosureCondition) expr, paramName); + sb.append(";\n"); + } else { + sb.append(" Object _v = "); + generateClosureExpr(sb, expr, paramName); + sb.append(";\n"); + sb.append(" return _v != null && !Boolean.FALSE.equals(_v);\n"); + } + } else { + for (final MALExpressionModel.ClosureStatement stmt : body) { + generateClosureStatement(sb, stmt, paramName); + } + sb.append(" return false;\n"); + } + sb.append("}\n"); + return sb.toString(); + } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java index 3ba103ed839b..455b5db1780e 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALExpressionModel.java @@ -399,7 +399,7 @@ public ClosureIndexAccess(final ClosureExpr index) { // ==================== Closure conditions ==================== - public interface ClosureCondition { + public interface ClosureCondition extends ClosureExpr { } @Getter diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java index ed8c7e04fbfb..2c216703871b 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALScriptParser.java @@ -108,6 +108,40 @@ public void syntaxError(final Recognizer recognizer, return new MALExprVisitor().visit(tree.additiveExpression()); } + /** + * Parse a standalone filter closure expression into a {@link ClosureArgument}. + * + * @param filterExpression e.g. {@code "{ tags -> tags.job_name == 'mysql-monitoring' }"} + */ + public static ClosureArgument parseFilter(final String filterExpression) { + final MALLexer lexer = new MALLexer(CharStreams.fromString(filterExpression)); + final CommonTokenStream tokens = new CommonTokenStream(lexer); + final MALParser parser = new MALParser(tokens); + + final List errors = new ArrayList<>(); + parser.removeErrorListeners(); + parser.addErrorListener(new BaseErrorListener() { + @Override + public void syntaxError(final Recognizer recognizer, + final Object offendingSymbol, + final int line, + final int charPositionInLine, + final String msg, + final RecognitionException e) { + errors.add(line + ":" + charPositionInLine + " " + msg); + } + }); + + final MALParser.FilterExpressionContext tree = parser.filterExpression(); + if (!errors.isEmpty()) { + throw new IllegalArgumentException( + "MAL filter expression parsing failed: " + String.join("; ", errors) + + " in expression: " + filterExpression); + } + + return new ClosureVisitor().visitClosure(tree.closureExpression()); + } + /** * Visitor transforming ANTLR4 parse tree into MAL expression AST. */ @@ -246,6 +280,11 @@ ClosureArgument visitClosure(final MALParser.ClosureExpressionContext ctx) { private List convertClosureBody( final MALParser.ClosureBodyContext ctx) { + // Bare condition or braced condition: { tags -> tags.x == 'v' } + if (ctx.closureCondition() != null) { + final ClosureCondition cond = convertCondition(ctx.closureCondition()); + return List.of(new ClosureExprStatement(cond)); + } final List stmts = new ArrayList<>(); for (final MALParser.ClosureStatementContext stmtCtx : ctx.closureStatement()) { stmts.add(convertClosureStatement(stmtCtx)); @@ -444,12 +483,14 @@ private ClosureMethodChain convertClosureMethodChain( final String target = ctx.closureTarget().IDENTIFIER().getText(); final List segments = new ArrayList<>(); - for (final MALParser.ClosureChainSegmentContext seg : ctx.closureChainSegment()) { - final boolean safeNav = segments.size() < ctx.closureChainSegment().size() - && ctx.safeNav().size() > 0; - // Determine if this segment uses safe navigation - // by checking position relative to safeNav tokens - segments.add(convertClosureChainSegment(seg, false)); + final int totalSegs = ctx.closureChainSegment().size(); + final int safeNavCount = ctx.safeNav().size(); + final int dotSegCount = totalSegs - safeNavCount; + + for (int i = 0; i < totalSegs; i++) { + final boolean isSafeNav = i >= dotSegCount; + segments.add(convertClosureChainSegment( + ctx.closureChainSegment().get(i), isSafeNav)); } return new ClosureMethodChain(target, segments); diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java index 2701f8337c7a..2081935639a7 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java @@ -22,8 +22,16 @@ import lombok.extern.slf4j.Slf4j; /** - * Same-FQCN replacement for upstream Expression. - * Wraps a compiled {@link MalExpression} (pure Java) instead of a Groovy DelegatingScript. + * Wraps a compiled {@link MalExpression} with runtime state management. + * + *

Two-phase usage: + *

    + *
  • {@link #parse()} — returns compile-time {@link ExpressionMetadata} extracted from the AST. + * Called once at startup by {@link org.apache.skywalking.oap.meter.analyzer.Analyzer#build} + * to discover sample names, scope type, aggregation labels, and metric type.
    • + *
  • {@link #run(Map)} — executes the compiled expression on actual sample data. + * Called at every ingestion cycle. Pure computation, no side effects.
    • + *
*/ @Slf4j @ToString(of = {"literal"}) diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java index b92318d9d372..320cc1e65644 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java @@ -17,54 +17,32 @@ package org.apache.skywalking.oap.meter.analyzer.dsl; -import java.io.IOException; -import java.io.InputStream; import java.util.HashMap; import java.util.Map; import java.util.Objects; -import java.util.Properties; -import java.util.concurrent.atomic.AtomicInteger; import lombok.ToString; import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALClassGenerator; /** - * Same-FQCN replacement for upstream FilterExpression. - * Loads transpiled {@link MalFilter} classes from mal-filter-expressions.properties - * manifest instead of Groovy filter closures -- no Groovy runtime needed. + * Compiles a MAL filter closure expression into a {@link MalFilter} + * using ANTLR4 parsing and Javassist bytecode generation. */ @Slf4j @ToString(of = {"literal"}) public class FilterExpression { - private static final String MANIFEST_PATH = "META-INF/mal-filter-expressions.properties"; - private static volatile Map FILTER_MAP; - private static final AtomicInteger LOADED_COUNT = new AtomicInteger(); + private static final MALClassGenerator GENERATOR = new MALClassGenerator(); private final String literal; private final MalFilter malFilter; - @SuppressWarnings("unchecked") public FilterExpression(final String literal) { this.literal = literal; - - final Map filterMap = loadManifest(); - final String className = filterMap.get(literal); - if (className == null) { - throw new IllegalStateException( - "Transpiled MAL filter not found for: " + literal - + ". Available filters: " + filterMap.size()); - } - try { - final Class filterClass = Class.forName(className); - malFilter = (MalFilter) filterClass.getDeclaredConstructor().newInstance(); - final int count = LOADED_COUNT.incrementAndGet(); - log.debug("Loaded transpiled MAL filter [{}/{}]: {}", count, filterMap.size(), literal); - } catch (ClassNotFoundException e) { - throw new IllegalStateException( - "Transpiled MAL filter class not found: " + className, e); - } catch (ReflectiveOperationException e) { + this.malFilter = GENERATOR.compileFilter(literal); + } catch (Exception e) { throw new IllegalStateException( - "Failed to instantiate transpiled MAL filter: " + className, e); + "Failed to compile MAL filter expression: " + literal, e); } } @@ -83,31 +61,4 @@ public Map filter(final Map sampleFa } return sampleFamilies; } - - private static Map loadManifest() { - if (FILTER_MAP != null) { - return FILTER_MAP; - } - synchronized (FilterExpression.class) { - if (FILTER_MAP != null) { - return FILTER_MAP; - } - final Map map = new HashMap<>(); - try (InputStream is = FilterExpression.class.getClassLoader().getResourceAsStream(MANIFEST_PATH)) { - if (is == null) { - log.warn("MAL filter manifest not found: {}", MANIFEST_PATH); - FILTER_MAP = map; - return map; - } - final Properties props = new Properties(); - props.load(is); - props.forEach((k, v) -> map.put((String) k, (String) v)); - } catch (IOException e) { - throw new IllegalStateException("Failed to load MAL filter manifest", e); - } - log.info("Loaded {} transpiled MAL filters from manifest", map.size()); - FILTER_MAP = map; - return map; - } - } } diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java index 2400cf319a4f..66737dfea78a 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java @@ -21,7 +21,6 @@ import java.util.Map; /** - * Pure Java replacement for Groovy-based MAL DelegatingScript. * Each compiled MAL expression implements this interface. */ public interface MalExpression { diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java index 9d8eaa259e92..c221fcba40e1 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java @@ -21,8 +21,7 @@ import java.util.Map; /** - * Pure Java replacement for Groovy Closure-based MAL filter expressions. - * Each transpiled filter expression implements this interface. + * Each compiled MAL filter expression implements this interface. */ @FunctionalInterface public interface MalFilter { diff --git a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java index 1c9747b56a25..752eb0f89864 100644 --- a/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java +++ b/oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java @@ -25,7 +25,7 @@ import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity; /** - * Pure Java functional interfaces replacing Groovy Closure parameters in SampleFamily methods. + * Functional interfaces used as parameters in {@link SampleFamily} methods. */ public final class SampleFamilyFunctions { @@ -33,7 +33,6 @@ private SampleFamilyFunctions() { } /** - * Replaces {@code Closure} in {@link SampleFamily#tag(groovy.lang.Closure)}. * Receives a mutable label map and returns the (possibly modified) map. */ @FunctionalInterface @@ -41,7 +40,6 @@ public interface TagFunction extends Function, Map} in {@link SampleFamily#filter(groovy.lang.Closure)}. * Tests whether a sample's labels match the filter criteria. */ @FunctionalInterface @@ -49,7 +47,6 @@ public interface SampleFilter extends Predicate> { } /** - * Replaces {@code Closure} in {@link SampleFamily#forEach(java.util.List, groovy.lang.Closure)}. * Called for each element in the array with the element value and a mutable labels map. */ @FunctionalInterface @@ -58,7 +55,6 @@ public interface ForEachFunction { } /** - * Replaces {@code Closure} in {@link SampleFamily#decorate(groovy.lang.Closure)}. * Decorates service meter entities. */ @FunctionalInterface @@ -66,9 +62,6 @@ public interface DecorateFunction extends Consumer { } /** - * Replaces {@code Closure>} in - * {@link SampleFamily#instance(java.util.List, String, java.util.List, String, - * org.apache.skywalking.oap.server.core.analysis.Layer, groovy.lang.Closure)}. * Extracts instance properties from sample labels. */ @FunctionalInterface diff --git a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java index 3e6cbbb8f50c..11b96642bb77 100644 --- a/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java +++ b/oap-server/analyzer/meter-analyzer/src/test/java/org/apache/skywalking/oap/meter/analyzer/compiler/MALClassGeneratorTest.java @@ -23,6 +23,8 @@ import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; class MALClassGeneratorTest { @@ -112,4 +114,56 @@ void generateSourceReturnsJavaCode() { org.junit.jupiter.api.Assertions.assertTrue( source.contains("getOrDefault")); } + + @Test + void filterSafeNavCompiles() throws Exception { + final String source = generator.generateFilterSource( + "{ tags -> tags.job_name == 'aws-cloud-eks-monitoring'" + + " && tags.Service?.trim() }"); + assertNotNull(source); + assertTrue(source.contains("trim"), "Generated source should contain trim()"); + assertNotNull(generator.compileFilter( + "{ tags -> tags.job_name == 'aws-cloud-eks-monitoring'" + + " && tags.Service?.trim() }")); + } + + // ==================== Error handling tests ==================== + + @Test + void emptyExpressionThrows() { + // Demo error: MAL expression parsing failed: 1:0 mismatched input '' + // expecting {IDENTIFIER, NUMBER, '(', '-'} + assertThrows(Exception.class, () -> generator.compile("test", "")); + } + + @Test + void malformedExpressionThrows() { + // Demo error: MAL expression parsing failed: 1:7 token recognition error at: '@' + assertThrows(Exception.class, + () -> generator.compile("test", "metric.@invalid")); + } + + @Test + void unclosedParenthesisThrows() { + // Demo error: MAL expression parsing failed: 1:8 mismatched input '' + // expecting {')', '+', '-', '*', '/'} + assertThrows(Exception.class, + () -> generator.compile("test", "(metric1 ")); + } + + @Test + void invalidFilterClosureThrows() { + // Demo error: MAL filter parsing failed: 1:0 mismatched input 'invalid' + // expecting '{' + assertThrows(Exception.class, + () -> generator.compileFilter("invalid filter")); + } + + @Test + void emptyFilterBodyThrows() { + // Demo error: MAL filter parsing failed: 1:1 mismatched input '}' + // expecting {IDENTIFIER, ...} + assertThrows(Exception.class, + () -> generator.compileFilter("{ }")); + } } diff --git a/oap-server/server-core/pom.xml b/oap-server/server-core/pom.xml index 056f3c9fa3d0..fd1214367152 100644 --- a/oap-server/server-core/pom.xml +++ b/oap-server/server-core/pom.xml @@ -107,10 +107,6 @@ io.zipkin.zipkin2 zipkin - - org.apache.groovy - groovy - diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java index 737e50ebed13..f84cb3cd9c29 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java @@ -22,7 +22,7 @@ import java.io.Reader; import java.util.HashMap; import java.util.Map; -import java.util.Objects; +import java.util.ServiceLoader; import java.util.function.BiFunction; import lombok.Getter; import lombok.extern.slf4j.Slf4j; @@ -35,12 +35,38 @@ import static java.util.stream.Collectors.toMap; +/** + * Loads hierarchy definitions from {@code hierarchy-definition.yml} and compiles + * matching rules into executable {@code BiFunction} + * matchers via a pluggable {@link HierarchyRuleProvider} (discovered through Java SPI). + * + *

Initialization (at startup, in CoreModuleProvider): + *

    + *
  1. Reads {@code hierarchy-definition.yml} containing three sections: + * {@code hierarchy} (layer-to-lower-layer mapping with rule names), + * {@code auto-matching-rules} (rule name to expression string), + * and {@code layer-levels} (layer to numeric level).
    2. + *
  2. Discovers a {@link HierarchyRuleProvider} via {@code ServiceLoader} + * (e.g., {@code CompiledHierarchyRuleProvider} from the hierarchy module).
    3. + *
  3. Calls {@link HierarchyRuleProvider#buildRules} which compiles each rule + * expression (e.g., {@code "{ (u, l) -> u.name == l.name }"}) into a + * {@code BiFunction} via ANTLR4 + Javassist.
    4. + *
  4. Wraps each compiled matcher in a {@link MatchingRule} and maps them + * to the layer hierarchy structure.
    5. + *
  5. Validates all layers exist in the {@code Layer} enum and that upper + * layers have higher level numbers than their lower layers.
    6. + *
+ * + *

The resulting {@link #getHierarchyDefinition()} map is consumed by + * {@link org.apache.skywalking.oap.server.core.hierarchy.HierarchyService} + * for runtime service matching. + */ @Slf4j public class HierarchyDefinitionService implements org.apache.skywalking.oap.server.library.module.Service { /** * Functional interface for building hierarchy matching rules. - * Implementations are provided by hierarchy-v1 (Groovy) or hierarchy-v2 (pure Java). + * Discovered via Java SPI ({@code ServiceLoader}). */ @FunctionalInterface public interface HierarchyRuleProvider { @@ -64,48 +90,29 @@ public HierarchyDefinitionService(final CoreModuleConfig moduleConfig, } /** - * Convenience constructor that uses the default Java rule provider. + * Convenience constructor that discovers a {@link HierarchyRuleProvider} + * via Java SPI ({@code ServiceLoader}). Only loads the provider when + * hierarchy is enabled. */ public HierarchyDefinitionService(final CoreModuleConfig moduleConfig) { - this(moduleConfig, new DefaultJavaRuleProvider()); + this.hierarchyDefinition = new HashMap<>(); + this.layerLevels = new HashMap<>(); + if (moduleConfig.isEnableHierarchy()) { + this.init(loadProvider()); + this.checkLayers(); + } } - /** - * Default pure Java rule provider with 4 built-in hierarchy matching rules. - * No Groovy dependency. - */ - private static class DefaultJavaRuleProvider implements HierarchyRuleProvider { - @Override - public Map> buildRules( - final Map ruleExpressions) { - final Map> registry = new HashMap<>(); - registry.put("name", (u, l) -> Objects.equals(u.getName(), l.getName())); - registry.put("short-name", (u, l) -> Objects.equals(u.getShortName(), l.getShortName())); - registry.put("lower-short-name-remove-ns", (u, l) -> { - final String sn = l.getShortName(); - final int dot = sn.lastIndexOf('.'); - return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); - }); - registry.put("lower-short-name-with-fqdn", (u, l) -> { - final String sn = u.getShortName(); - final int colon = sn.lastIndexOf(':'); - return colon > 0 && Objects.equals( - sn.substring(0, colon), - l.getShortName() + ".svc.cluster.local"); - }); - - final Map> rules = new HashMap<>(); - ruleExpressions.forEach((name, expression) -> { - final BiFunction fn = registry.get(name); - if (fn == null) { - throw new IllegalArgumentException( - "Unknown hierarchy matching rule: " + name - + ". Known rules: " + registry.keySet()); - } - rules.put(name, fn); - }); - return rules; + private static HierarchyRuleProvider loadProvider() { + final ServiceLoader loader = + ServiceLoader.load(HierarchyRuleProvider.class); + for (final HierarchyRuleProvider provider : loader) { + log.info("Using hierarchy rule provider: {}", provider.getClass().getName()); + return provider; } + throw new IllegalStateException( + "No HierarchyRuleProvider found on classpath. " + + "Ensure the hierarchy analyzer module is included."); } @SuppressWarnings("unchecked") diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java index 5eaa22752e1e..6f18f28995ea 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java @@ -38,6 +38,33 @@ import org.apache.skywalking.oap.server.library.module.ModuleManager; import org.apache.skywalking.oap.server.library.util.RunnableWithExceptionProtection; +/** + * Runtime service that builds hierarchy relations between services and instances. + * + *

Uses the compiled matching rules from + * {@link HierarchyDefinitionService} to determine if two services are + * hierarchically related (e.g., a MESH service sitting above a K8S_SERVICE). + * + *

Two paths for creating relations: + *

    + *
  1. Explicit (from agent telemetry): receivers call + * {@link #toServiceHierarchyRelation} or {@link #toInstanceHierarchyRelation} + * when agents report detected service-to-service relationships.
    2. + *
  2. Auto-matching (scheduled background task): {@link #startAutoMatchingServiceHierarchy()} + * starts a background task that runs every 20 seconds, comparing all known + * service pairs against the compiled hierarchy rules: + *
      + *
    • Retrieves all services from {@link MetadataQueryService}.
      • + *
    • For each pair (i, j), checks if hierarchy rules exist for + * layer[i]→layer[j] or layer[j]→layer[i].
      • + *
    • Invokes {@link HierarchyDefinitionService.MatchingRule#match} + * which executes the compiled {@code BiFunction}.
      • + *
    • If matched, creates a {@link ServiceHierarchyRelation} and sends it + * to {@link SourceReceiver} for persistence.
      • + *
    + *
    3. + *
+ */ @Slf4j public class HierarchyService implements org.apache.skywalking.oap.server.library.module.Service { private final ModuleManager moduleManager; diff --git a/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java b/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java index b5be35e468fb..c859bf1e6d55 100644 --- a/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java +++ b/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java @@ -28,6 +28,7 @@ import org.apache.skywalking.oap.server.library.util.ResourceUtils; import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; +import org.apache.skywalking.oap.server.core.config.compiler.CompiledHierarchyRuleProvider; import org.yaml.snakeyaml.Yaml; import static org.junit.jupiter.api.Assertions.assertEquals; @@ -70,7 +71,7 @@ Collection allRulesProduceIdenticalResults() throws FileNotFoundExc final Map ruleExpressions = (Map) config.get("auto-matching-rules"); final GroovyHierarchyRuleProvider groovyProvider = new GroovyHierarchyRuleProvider(); - final JavaHierarchyRuleProvider javaProvider = new JavaHierarchyRuleProvider(); + final CompiledHierarchyRuleProvider javaProvider = new CompiledHierarchyRuleProvider(); final Map> v1Rules = groovyProvider.buildRules(ruleExpressions); diff --git a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java index c428c26d2b07..2abd32cdf6c7 100644 --- a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java +++ b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java @@ -23,17 +23,13 @@ import java.util.Collection; import java.util.List; import java.util.Map; +import org.apache.skywalking.oap.log.analyzer.compiler.LALClassGenerator; import org.apache.skywalking.oap.log.analyzer.dsl.LalExpression; -import org.apache.skywalking.oap.server.checker.InMemoryCompiler; -import org.apache.skywalking.oap.server.transpiler.lal.LalToJavaTranspiler; -import org.junit.jupiter.api.AfterAll; -import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import org.yaml.snakeyaml.Yaml; import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertTrue; import static org.junit.jupiter.api.Assertions.fail; /** @@ -41,32 +37,14 @@ * For each LAL rule across all LAL YAML files: *
    *
  • Path A (v1): Verify Groovy compiles the DSL without error
    • - *
  • Path B (v2): Transpile to Java, compile in-memory, verify it - * implements {@link LalExpression}
    • + *
  • Path B (v2): ANTLR4 + Javassist compilation via {@link LALClassGenerator}, + * verify it produces a valid {@link LalExpression}
    • *
- * Both paths must accept the same DSL input. The transpiled Java class - * must compile and be instantiable. */ class LalComparisonTest { - private static InMemoryCompiler COMPILER; - private static int CLASS_COUNTER; - - @BeforeAll - static void initCompiler() throws Exception { - COMPILER = new InMemoryCompiler(); - CLASS_COUNTER = 0; - } - - @AfterAll - static void closeCompiler() throws Exception { - if (COMPILER != null) { - COMPILER.close(); - } - } - @TestFactory - Collection lalScriptsTranspileAndCompile() throws Exception { + Collection lalScriptsCompile() throws Exception { final List tests = new ArrayList<>(); final Map> yamlRules = loadAllLalYamlFiles(); @@ -75,7 +53,7 @@ Collection lalScriptsTranspileAndCompile() throws Exception { for (final LalRule rule : entry.getValue()) { tests.add(DynamicTest.dynamicTest( yamlFile + " | " + rule.name, - () -> verifyTranspileAndCompile(rule.name, rule.dsl) + () -> verifyCompilation(rule.name, rule.dsl) )); } } @@ -83,8 +61,8 @@ Collection lalScriptsTranspileAndCompile() throws Exception { return tests; } - private void verifyTranspileAndCompile(final String ruleName, - final String dsl) throws Exception { + private void verifyCompilation(final String ruleName, + final String dsl) throws Exception { // ---- V1: Verify Groovy can parse the DSL ---- try { final groovy.lang.GroovyShell sh = new groovy.lang.GroovyShell(); @@ -95,22 +73,11 @@ private void verifyTranspileAndCompile(final String ruleName, return; } - // ---- V2: Transpile and compile ---- + // ---- V2: ANTLR4 + Javassist compilation ---- try { - final LalToJavaTranspiler transpiler = new LalToJavaTranspiler(); - final String className = "LalExpr_check_" + (CLASS_COUNTER++); - final String javaSource = transpiler.transpile(className, dsl); - assertNotNull(javaSource, "V2 transpiler should produce source for '" + ruleName + "'"); - - final Class clazz = COMPILER.compile( - LalToJavaTranspiler.GENERATED_PACKAGE, className, javaSource); - - assertTrue(LalExpression.class.isAssignableFrom(clazz), - "Generated class should implement LalExpression for '" + ruleName + "'"); - - final LalExpression expr = (LalExpression) clazz - .getDeclaredConstructor().newInstance(); - assertNotNull(expr, "V2 should instantiate for '" + ruleName + "'"); + final LALClassGenerator generator = new LALClassGenerator(); + final LalExpression expr = generator.compile(dsl); + assertNotNull(expr, "V2 should compile '" + ruleName + "'"); } catch (Exception e) { fail("V2 (Java) failed for LAL rule '" + ruleName + "': " + e.getMessage()); } @@ -161,15 +128,15 @@ private Map> loadAllLalYamlFiles() throws Exception { } private Path findResourceDir(final String name) { - final Path starterResources = Path.of( - "oap-server/server-starter/src/main/resources/" + name); - if (Files.isDirectory(starterResources)) { - return starterResources; - } - final Path fromRoot = Path.of( - System.getProperty("user.dir")).resolve("../../server-starter/src/main/resources/" + name); - if (Files.isDirectory(fromRoot)) { - return fromRoot; + final String[] candidates = { + "oap-server/server-starter/src/main/resources/" + name, + "../../../oap-server/server-starter/src/main/resources/" + name + }; + for (final String candidate : candidates) { + final Path path = Path.of(candidate); + if (Files.isDirectory(path)) { + return path; + } } return null; } diff --git a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java index 5a6a5ac93a34..3b6117217523 100644 --- a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java +++ b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java @@ -26,16 +26,13 @@ import java.util.List; import java.util.Map; import java.util.concurrent.atomic.AtomicInteger; -import com.google.common.collect.ImmutableMap; import lombok.extern.slf4j.Slf4j; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALClassGenerator; import org.apache.skywalking.oap.meter.analyzer.dsl.DSL; import org.apache.skywalking.oap.meter.analyzer.dsl.Expression; +import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionMetadata; import org.apache.skywalking.oap.meter.analyzer.dsl.ExpressionParsingContext; import org.apache.skywalking.oap.meter.analyzer.dsl.MalExpression; -import org.apache.skywalking.oap.server.checker.InMemoryCompiler; -import org.apache.skywalking.oap.server.transpiler.mal.MalToJavaTranspiler; -import org.junit.jupiter.api.AfterAll; -import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import org.yaml.snakeyaml.Yaml; @@ -46,34 +43,16 @@ * Dual-path comparison test for MAL (Meter Analysis Language) expressions. * For each metric rule across all MAL YAML files: *
    - *
  • Path A (v1): Groovy compilation via upstream {@link DSL#parse(String, String)}
    • - *
  • Path B (v2): Transpiled Java via {@link MalToJavaTranspiler}, compiled in-memory
    • + *
  • Path A (v1): Groovy compilation via upstream {@code DSL.parse()}
    • + *
  • Path B (v2): ANTLR4 + Javassist compilation via {@link MALClassGenerator}
    • *
- * Both paths run {@code parse()} with empty input and compare the resulting - * {@link ExpressionParsingContext} (samples, scope, downsampling, aggregation labels). + * Both paths run metadata extraction and compare the resulting metadata + * (samples, scope, downsampling, aggregation labels). */ @Slf4j class MalComparisonTest { - private static InMemoryCompiler COMPILER; - private static final AtomicInteger CLASS_COUNTER = new AtomicInteger(); - private static final AtomicInteger V2_TRANSPILE_GAPS = new AtomicInteger(); - - @BeforeAll - static void initCompiler() throws Exception { - COMPILER = new InMemoryCompiler(); - } - - @AfterAll - static void closeCompiler() throws Exception { - if (COMPILER != null) { - COMPILER.close(); - } - final int gaps = V2_TRANSPILE_GAPS.get(); - if (gaps > 0) { - log.warn("{} MAL expressions could not be transpiled to Java (known transpiler gaps)", gaps); - } - } + private static final AtomicInteger V2_COMPILE_GAPS = new AtomicInteger(); @TestFactory Collection malExpressionsMatch() throws Exception { @@ -97,70 +76,49 @@ private void compareExpression(final String metricName, final String expression) throws Exception { // ---- V1: Groovy path ---- ExpressionParsingContext v1Ctx = null; - String v1Error = null; try { final Expression v1Expr = DSL.parse(metricName, expression); v1Ctx = v1Expr.parse(); } catch (Exception e) { - v1Error = e.getMessage(); + // V1 failed - skip comparison } - // ---- V2: Transpiled Java path ---- - ExpressionParsingContext v2Ctx = null; + // ---- V2: ANTLR4 + Javassist compilation ---- + ExpressionMetadata v2Meta = null; String v2Error = null; try { - final MalToJavaTranspiler transpiler = new MalToJavaTranspiler(); - final String className = "MalExpr_check_" + CLASS_COUNTER.getAndIncrement(); - final String javaSource = transpiler.transpileExpression(className, expression); - - final Class clazz = COMPILER.compile( - MalToJavaTranspiler.GENERATED_PACKAGE, className, javaSource); - final MalExpression malExpr = (MalExpression) clazz - .getDeclaredConstructor().newInstance(); - - // Run parse: create parsing context, execute with empty map, extract context - try (ExpressionParsingContext ctx = ExpressionParsingContext.create()) { - try { - malExpr.run(ImmutableMap.of()); - } catch (Exception ignored) { - // Expected: expressions fail with empty input - } - ctx.validate(expression); - v2Ctx = ctx; - } + final MALClassGenerator generator = new MALClassGenerator(); + final MalExpression malExpr = generator.compile(metricName, expression); + v2Meta = malExpr.metadata(); } catch (Exception e) { v2Error = e.getMessage(); } // ---- Compare ---- - if (v1Ctx == null && v2Ctx == null) { - // Both failed - acceptable (known limitations in both paths) + if (v1Ctx == null && v2Meta == null) { return; } if (v1Ctx == null) { - // V1 failed but V2 succeeded - V2 is more capable, acceptable return; } - if (v2Ctx == null) { - // V2 transpiler/compilation gap - log and count, not a test failure. - // These are known limitations of the transpiler that will be addressed incrementally. - V2_TRANSPILE_GAPS.incrementAndGet(); - log.info("V2 transpile gap for '{}': {}", metricName, v2Error); + if (v2Meta == null) { + V2_COMPILE_GAPS.incrementAndGet(); + log.info("V2 compile gap for '{}': {}", metricName, v2Error); return; } - // Both succeeded - compare contexts - assertEquals(v1Ctx.getSamples(), v2Ctx.getSamples(), + // Both succeeded - compare metadata + assertEquals(v1Ctx.getSamples(), v2Meta.getSamples(), metricName + ": samples mismatch"); - assertEquals(v1Ctx.getScopeType(), v2Ctx.getScopeType(), + assertEquals(v1Ctx.getScopeType(), v2Meta.getScopeType(), metricName + ": scopeType mismatch"); - assertEquals(v1Ctx.getDownsampling(), v2Ctx.getDownsampling(), + assertEquals(v1Ctx.getDownsampling(), v2Meta.getDownsampling(), metricName + ": downsampling mismatch"); - assertEquals(v1Ctx.isHistogram(), v2Ctx.isHistogram(), + assertEquals(v1Ctx.isHistogram(), v2Meta.isHistogram(), metricName + ": isHistogram mismatch"); - assertEquals(v1Ctx.getScopeLabels(), v2Ctx.getScopeLabels(), + assertEquals(v1Ctx.getScopeLabels(), v2Meta.getScopeLabels(), metricName + ": scopeLabels mismatch"); - assertEquals(v1Ctx.getAggregationLabels(), v2Ctx.getAggregationLabels(), + assertEquals(v1Ctx.getAggregationLabels(), v2Meta.getAggregationLabels(), metricName + ": aggregationLabels mismatch"); } @@ -240,17 +198,15 @@ private void collectYamlFiles(final File dir, final String prefix, } private Path findResourceDir(final String name) { - // Look in server-starter resources - final Path starterResources = Path.of( - "oap-server/server-starter/src/main/resources/" + name); - if (Files.isDirectory(starterResources)) { - return starterResources; - } - // Try from project root - final Path fromRoot = Path.of( - System.getProperty("user.dir")).resolve("../../server-starter/src/main/resources/" + name); - if (Files.isDirectory(fromRoot)) { - return fromRoot; + final String[] candidates = { + "oap-server/server-starter/src/main/resources/" + name, + "../../../oap-server/server-starter/src/main/resources/" + name + }; + for (final String candidate : candidates) { + final Path path = Path.of(candidate); + if (Files.isDirectory(path)) { + return path; + } } return null; } diff --git a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java index 2c3ee71667d2..4400701691ca 100644 --- a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java +++ b/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java @@ -29,11 +29,8 @@ import java.util.Set; import groovy.lang.Closure; import groovy.lang.GroovyShell; +import org.apache.skywalking.oap.meter.analyzer.compiler.MALClassGenerator; import org.apache.skywalking.oap.meter.analyzer.dsl.MalFilter; -import org.apache.skywalking.oap.server.checker.InMemoryCompiler; -import org.apache.skywalking.oap.server.transpiler.mal.MalToJavaTranspiler; -import org.junit.jupiter.api.AfterAll; -import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import org.yaml.snakeyaml.Yaml; @@ -46,28 +43,12 @@ * For each unique filter expression across all MAL YAML files: *
    *
  • Path A (v1): Groovy {@code GroovyShell.evaluate()} -> {@code Closure}
    • - *
  • Path B (v2): Transpile to {@link MalFilter}, compile in-memory
    • + *
  • Path B (v2): ANTLR4 + Javassist compilation via {@link MALClassGenerator}
    • *
* Both paths are invoked with representative tag maps and results compared. */ class MalFilterComparisonTest { - private static InMemoryCompiler COMPILER; - private static int CLASS_COUNTER; - - @BeforeAll - static void initCompiler() throws Exception { - COMPILER = new InMemoryCompiler(); - CLASS_COUNTER = 0; - } - - @AfterAll - static void closeCompiler() throws Exception { - if (COMPILER != null) { - COMPILER.close(); - } - } - @TestFactory Collection filterExpressionsMatch() throws Exception { final Set filters = collectAllFilterExpressions(); @@ -85,7 +66,6 @@ Collection filterExpressionsMatch() throws Exception { @SuppressWarnings("unchecked") private void compareFilter(final String filterExpr) throws Exception { - // Extract the tag key from the filter expression for test data final List> testTags = buildTestTags(filterExpr); // ---- V1: Groovy closure ---- @@ -97,16 +77,11 @@ private void compareFilter(final String filterExpr) throws Exception { return; } - // ---- V2: Transpiled MalFilter ---- + // ---- V2: ANTLR4 + Javassist compilation ---- final MalFilter v2Filter; try { - final MalToJavaTranspiler transpiler = new MalToJavaTranspiler(); - final String className = "MalFilter_check_" + (CLASS_COUNTER++); - final String javaSource = transpiler.transpileFilter(className, filterExpr); - - final Class clazz = COMPILER.compile( - MalToJavaTranspiler.GENERATED_PACKAGE, className, javaSource); - v2Filter = (MalFilter) clazz.getDeclaredConstructor().newInstance(); + final MALClassGenerator generator = new MALClassGenerator(); + v2Filter = generator.compileFilter(filterExpr); } catch (Exception e) { fail("V2 (Java) failed for filter: " + filterExpr + " - " + e.getMessage()); return; @@ -118,14 +93,12 @@ private void compareFilter(final String filterExpr) throws Exception { try { v1Result = v1Closure.call(tags); } catch (Exception e) { - // Some filters error on empty/missing tags in Groovy too continue; } boolean v2Result; try { v2Result = v2Filter.test(tags); } catch (NullPointerException e) { - // List.of().contains(null) throws NPE; Groovy 'in' returns false v2Result = false; } assertEquals(v1Result, v2Result, @@ -137,12 +110,8 @@ private void compareFilter(final String filterExpr) throws Exception { private List> buildTestTags(final String filterExpr) { final List> testTags = new ArrayList<>(); - // Always test with an empty map testTags.add(new HashMap<>()); - // Extract key-value patterns from the expression to build matching and non-matching tags. - // Common patterns: tags.job_name == 'mysql-monitoring', tags.Namespace == 'AWS/DynamoDB' - // We build: one matching map, one non-matching map final java.util.regex.Pattern kvPattern = java.util.regex.Pattern.compile("tags\\.(\\w+)\\s*==\\s*'([^']+)'"); final java.util.regex.Matcher matcher = kvPattern.matcher(filterExpr); @@ -161,7 +130,6 @@ private List> buildTestTags(final String filterExpr) { testTags.add(mismatchTags); } - // Also test with a random unrelated key final Map unrelatedTags = new HashMap<>(); unrelatedTags.put("unrelated_key", "some_value"); testTags.add(unrelatedTags); @@ -183,7 +151,6 @@ private Set collectAllFilterExpressions() throws Exception { collectFiltersFromDir(dirPath.toFile(), yaml, filters); } - // Also check log-mal-rules and envoy-metrics-rules for (final String dir : new String[]{"log-mal-rules", "envoy-metrics-rules"}) { final Path dirPath = findResourceDir(dir); if (dirPath != null) { @@ -225,15 +192,15 @@ private void collectFiltersFromDir(final File dir, final Yaml yaml, } private Path findResourceDir(final String name) { - final Path starterResources = Path.of( - "oap-server/server-starter/src/main/resources/" + name); - if (Files.isDirectory(starterResources)) { - return starterResources; - } - final Path fromRoot = Path.of( - System.getProperty("user.dir")).resolve("../../server-starter/src/main/resources/" + name); - if (Files.isDirectory(fromRoot)) { - return fromRoot; + final String[] candidates = { + "oap-server/server-starter/src/main/resources/" + name, + "../../../oap-server/server-starter/src/main/resources/" + name + }; + for (final String candidate : candidates) { + final Path path = Path.of(candidate); + if (Files.isDirectory(path)) { + return path; + } } return null; } diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java index cfaf1d5beee5..66737dfea78a 100644 --- a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java +++ b/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java @@ -21,10 +21,14 @@ import java.util.Map; /** - * Pure Java replacement for Groovy-based MAL DelegatingScript. - * Each transpiled MAL expression implements this interface. + * Each compiled MAL expression implements this interface. */ -@FunctionalInterface public interface MalExpression { SampleFamily run(Map samples); + + /** + * Returns compile-time metadata extracted from the expression AST: + * sample names, scope type, aggregation labels, downsampling, etc. + */ + ExpressionMetadata metadata(); } From f4127038b610a02cd31a5d3b6deef3071b51ac21 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 1 Mar 2026 10:24:42 +0800 Subject: [PATCH 10/64] Replace groovy-replacement-plan.md with dsl-compiler-design.md covering all four DSL compilers (OAL, MAL, LAL, Hierarchy). Remove Groovy references from docs: LAL code blocks, hierarchy matching rule labels, and stale MeterProcessor comment. Co-Authored-By: Claude Opus 4.6 --- docs/en/academy/dsl-compiler-design.md | 157 +++ docs/en/academy/groovy-replacement-plan.md | 1025 ----------------- docs/en/concepts-and-designs/lal.md | 34 +- .../service-hierarchy-configuration.md | 2 +- .../concepts-and-designs/service-hierarchy.md | 46 +- docs/menu.yml | 2 + .../meter/process/MeterProcessor.java | 2 +- 7 files changed, 200 insertions(+), 1068 deletions(-) create mode 100644 docs/en/academy/dsl-compiler-design.md delete mode 100644 docs/en/academy/groovy-replacement-plan.md diff --git a/docs/en/academy/dsl-compiler-design.md b/docs/en/academy/dsl-compiler-design.md new file mode 100644 index 000000000000..ce14082e2084 --- /dev/null +++ b/docs/en/academy/dsl-compiler-design.md @@ -0,0 +1,157 @@ +# DSL Compiler Design: ANTLR4 + Javassist + +## Overview + +SkyWalking OAP server uses four domain-specific languages (DSLs) for telemetry analysis. +All four share the same compilation tech stack: **ANTLR4** for grammar parsing and **Javassist** for +runtime bytecode generation. + +| DSL | Purpose | Input | Generated Output | +|-----|---------|-------|-----------------| +| **OAL** (Observability Analysis Language) | Trace/mesh metrics aggregation | `.oal` script files | Metrics classes, builders, dispatchers | +| **MAL** (Meter Analysis Language) | Meter/metrics expression evaluation | YAML config `exp` fields | `MalExpression` implementations | +| **LAL** (Log Analysis Language) | Log processing pipelines | YAML config `filter` blocks | `LalExpression` implementations | +| **Hierarchy Matching Rules** | Service hierarchy relationship matching | YAML config expressions | `BiFunction` implementations | + +## Compilation Pipeline + +All four DSLs follow the same three-phase compilation pipeline at OAP startup: + +``` +DSL string (from .oal script or YAML config) + | + v +Phase 1: ANTLR4 Parsing + Lexer + Parser (generated from .g4 grammars at build time) + → Immutable AST model + | + v +Phase 2: Java Source Generation + Walk AST model, emit Java source code as strings + | + v +Phase 3: Javassist Bytecode Generation + ClassPool.makeClass() → CtClass → addMethod(source) → toClass() + → Ready-to-use class instance loaded into JVM +``` + +### What Each DSL Generates + +| DSL | Interface / Base Class | Key Method | +|-----|----------------------|------------| +| OAL | Extends metrics function class (e.g., `LongAvgMetrics`) | `id()`, `serialize()`, `deserialize()`, plus dispatcher `dispatch(source)` | +| MAL metric | `MalExpression` | `SampleFamily run(Map samples)` | +| MAL filter | `Predicate>` | `boolean test(Map tags)` | +| LAL | `LalExpression` | `void execute(Object filterSpec, Object binding)` | +| Hierarchy | `BiFunction` | `Boolean apply(Service upper, Service lower)` | + +OAL is the most complex -- it generates **three classes per metric** (metrics class with storage annotations, +metrics builder for serialization, and source dispatcher for routing), whereas MAL/LAL/Hierarchy each generate +a single functional class per expression. + +## ANTLR4 Grammars + +Each DSL has its own ANTLR4 lexer and parser grammar. The Maven ANTLR4 plugin generates Java lexer/parser +classes at build time; these are then used at runtime to parse DSL strings. + +| DSL | Grammar Location | +|-----|-----------------| +| OAL | `oap-server/oal-grammar/src/main/antlr4/.../OALLexer.g4`, `OALParser.g4` | +| MAL | `oap-server/analyzer/meter-analyzer/src/main/antlr4/.../MALLexer.g4`, `MALParser.g4` | +| LAL | `oap-server/analyzer/log-analyzer/src/main/antlr4/.../LALLexer.g4`, `LALParser.g4` | +| Hierarchy | `oap-server/analyzer/hierarchy/src/main/antlr4/.../HierarchyRuleLexer.g4`, `HierarchyRuleParser.g4` | + +## Javassist Constraints + +Javassist compiles Java source strings into bytecode but has limitations that shape the code generation: + +- **No anonymous inner classes or lambdas** -- Callback-based APIs (e.g., LAL's `filterSpec.extractor(Consumer)`) + require pre-compiling each callback as a separate `CtClass` implementing the needed interface. +- **No generics in method bodies** -- Generated source uses raw types with explicit casts. +- **Class loading anchor** -- Each DSL uses a `PackageHolder` marker class so that + `ctClass.toClass(PackageHolder.class)` loads the generated class into the correct module/package + (required for JDK 9+ module system). + +OAL additionally uses **FreeMarker templates** to generate method bodies for metrics classes, builders, and +dispatchers, since these classes are more complex and benefit from template-driven generation. + +## Module Structure + +``` +oap-server/ + oal-grammar/ # OAL: ANTLR4 grammar + oal-rt/ # OAL: compiler + runtime (Javassist + FreeMarker) + analyzer/ + meter-analyzer/ # MAL: grammar + compiler + runtime + log-analyzer/ # LAL: grammar + compiler + runtime + hierarchy/ # Hierarchy: grammar + compiler + runtime + agent-analyzer/ # Calls MAL compiler for meter data +``` + +OAL keeps grammar and runtime in separate modules (`oal-grammar` and `oal-rt`) because `server-core` +depends on the grammar while the runtime implementation depends on `server-core` (avoiding circular +dependency). MAL, LAL, and Hierarchy are each self-contained in a single module. + +## Groovy Replacement (MAL, LAL, Hierarchy) + +Reference: [Discussion #13716](https://github.com/apache/skywalking/discussions/13716) + +MAL, LAL, and Hierarchy previously used **Groovy** as the runtime scripting engine. OAL has always used +ANTLR4 + Javassist. The Groovy-based DSLs were replaced for the following reasons: + +1. **Startup cost** -- 1,250+ `GroovyShell.parse()` calls at OAP boot, each spinning up the full Groovy + compiler pipeline. + +2. **Runtime execution overhead** -- MAL expressions execute on every metrics ingestion cycle. Per-expression + overhead from dynamic Groovy compounds at scale: property resolution through 4+ layers of indirection, + `ExpandoMetaClass` closure allocation for simple arithmetic, and megamorphic call sites that defeat JIT + optimization. + +3. **Late error detection** -- MAL uses dynamic Groovy; typos in metric names or invalid method chains are + only discovered when that specific expression runs with real data. + +4. **Debugging complexity** -- Stack traces include Groovy MOP internals (`CallSite`, `MetaClassImpl`, + `ExpandoMetaClass`), obscuring the actual expression logic. + +5. **GraalVM incompatibility** -- `invokedynamic` bootstrapping and `ExpandoMetaClass` are fundamentally + incompatible with ahead-of-time (AOT) compilation, blocking the + [GraalVM native-image distribution](https://github.com/apache/skywalking-graalvm-distro). + +The DSL grammar for users remains **100% unchanged** -- the same expressions written in YAML config files +work exactly as before. Only the internal compilation engine was replaced. + +### Verification: Groovy v1 Checker + +To ensure the new Java compilers produce identical results to the original Groovy implementation, +a **dual-path comparison test suite** is maintained under `test/script-compiler/`: + +``` +test/script-compiler/ + mal-groovy/ # MAL v1: original Groovy-based implementation + lal-groovy/ # LAL v1: original Groovy-based implementation + hierarchy-groovy/ # Hierarchy v1: original Groovy-based implementation + mal-lal-v1-v2-checker/ # Runs every MAL/LAL expression through BOTH v1 and v2, compares results + hierarchy-v1-v2-checker/ # Runs every hierarchy rule through BOTH v1 and v2, compares results +``` + +The checker mechanism: + +1. Loads all production YAML config files (the same files used by OAP at runtime) +2. For each DSL expression, compiles with **both** v1 (Groovy) and v2 (ANTLR4 + Javassist) +3. Compares the compiled artifacts: + - **MAL**: Compare generated Java source code, extracted metadata (sample names, aggregation labels, + downsampling type, percentile config), and execution results with sample data + - **LAL**: Compare compiled expression execution against FilterSpec/Binding mocks + - **Hierarchy**: Compare `BiFunction` evaluation with test Service pairs + +This ensures 100% behavioral parity. The Groovy v1 modules are **test-only dependencies** -- they are not +included in the OAP distribution. + +#### Current Checker Results + +| Checker | Expressions Tested | Status | +|---------|-------------------|--------| +| MAL metric expressions | 1,187 | All pass | +| MAL filter expressions | 29 | All pass | +| LAL scripts | 10 | All pass | +| Hierarchy rules | 22 | All pass | diff --git a/docs/en/academy/groovy-replacement-plan.md b/docs/en/academy/groovy-replacement-plan.md deleted file mode 100644 index f656a7b94d69..000000000000 --- a/docs/en/academy/groovy-replacement-plan.md +++ /dev/null @@ -1,1025 +0,0 @@ -# Groovy Replacement Plan: Build-Time Transpiler for MAL, LAL, and Hierarchy Scripts - -Reference: [Discussion #13716](https://github.com/apache/skywalking/discussions/13716) -Reference Implementation: [skywalking-graalvm-distro](https://github.com/apache/skywalking-graalvm-distro) - -## 1. Background and Motivation - -SkyWalking OAP server currently uses Groovy as the runtime scripting engine for three subsystems: - -| Subsystem | YAML Files | Expressions | Groovy Pattern | -|-----------|-----------|-------------|----------------| -| MAL (Meter Analysis Language) | 71 (11 meter-analyzer-config, 55 otel-rules, 2 log-mal-rules, 2 envoy-metrics-rules, 1 telegraf-rules) | 1,254 metric + 29 filter | Dynamic Groovy: `propertyMissing()`, `ExpandoMetaClass` on `Number`, closures | -| LAL (Log Analysis Language) | 8 | 10 rules | `@CompileStatic` Groovy: delegation-based closure DSL, safe navigation (`?.`), `as` casts | -| Hierarchy Matching | 1 (hierarchy-definition.yml) | 4 rules | `GroovyShell.evaluate()` for `Closure` | - -### Problems with Groovy Runtime - -1. **Startup Cost**: 1,250+ `GroovyShell.parse()` calls at OAP boot, each spinning up the full Groovy compiler pipeline. -2. **Runtime Errors Instead of Compile-Time Errors**: MAL uses dynamic Groovy -- typos in metric names or invalid method chains are only discovered when that specific expression runs with real data. -3. **Debugging Complexity**: Stack traces include Groovy MOP internals (`CallSite`, `MetaClassImpl`, `ExpandoMetaClass`), obscuring the actual expression logic. -4. **Runtime Execution Performance (Most Critical)**: MAL expressions execute on every metrics ingestion cycle. Per-expression overhead from dynamic Groovy compounds at scale: - - Property resolution: `CallSite` -> `MetaClassImpl.invokePropertyOrMissing()` -> `ExpressionDelegate.propertyMissing()` -> `ThreadLocal` lookup (4+ layers of indirection per metric name lookup) - - Method calls: Groovy `CallSite` dispatch with MetaClass lookup and MOP interception checks - - Arithmetic (`metric * 1000`): `ExpandoMetaClass` closure allocation + metaclass lookup + dynamic dispatch for what Java does as a single `imul` - - Per ingestion cycle: ~1,250 `propertyMissing()` calls, ~3,750 MOP method dispatches, ~29 metaclass arithmetic ops, ~200 closure allocations - - JIT cannot optimize Groovy's megamorphic call sites, defeating inlining and branch prediction -5. **GraalVM Incompatibility**: `invokedynamic` bootstrapping and `ExpandoMetaClass` are fundamentally incompatible with AOT compilation. - -### Goal - -Eliminate Groovy from the OAP runtime entirely. Groovy becomes a **build-time-only** dependency used solely for AST parsing by the transpiler. Zero `GroovyShell`, zero `ExpandoMetaClass`, zero MOP at runtime. - ---- - -## 2. Solution Architecture - -### Build-Time Transpiler: Groovy DSL -> Pure Java Source Code - -``` -BUILD TIME (Maven compile phase): - MAL YAML files (71 files, 1,250+ expressions) - LAL YAML files (8 files, 10 scripts) - | - v - MalToJavaTranspiler / LalToJavaTranspiler - (Groovy CompilationUnit at CONVERSION phase -- AST parsing only, no execution) - | - v - ~1,254 MalExpr_*.java + ~6 LalExpr_*.java + MalFilter_*.java - | - v - javax.tools.JavaCompiler -> .class files on classpath - META-INF/mal-expressions.txt (manifest) - META-INF/mal-filter-expressions.properties (manifest) - META-INF/lal-expressions.txt (manifest) - -RUNTIME (OAP Server): - Class.forName(className) -> MalExpression / LalExpression instance - Zero Groovy. Zero GroovyShell. Zero ExpandoMetaClass. -``` - -The transpiler approach is already fully implemented and validated in the [skywalking-graalvm-distro](https://github.com/apache/skywalking-graalvm-distro) repository. - ---- - -## 3. Detailed Design - -### 3.1 New Functional Interfaces - -Three core interfaces replace Groovy's `DelegatingScript` and `Closure`: - -```java -// MAL: replaces DelegatingScript + ExpandoMetaClass + ExpressionDelegate.propertyMissing() -@FunctionalInterface -public interface MalExpression { - SampleFamily run(Map samples); -} - -// MAL: replaces Closure from GroovyShell.evaluate() for filter expressions -@FunctionalInterface -public interface MalFilter { - boolean test(Map tags); -} - -// LAL: replaces LALDelegatingScript + @CompileStatic closure DSL -@FunctionalInterface -public interface LalExpression { - void execute(FilterSpec filterSpec, Binding binding); -} -``` - -### 3.2 SampleFamily: Closure -> Functional Interface - -Five `SampleFamily` methods currently accept `groovy.lang.Closure`. Each gets a new overload with a Java functional interface. During transition both overloads coexist; eventually the Closure overloads are removed. - -| Method | Current (Groovy) | New (Java) | Functional Interface | -|--------|-----------------|------------|---------------------| -| `tag()` | `Closure` with `tags.key = val` | `TagFunction` | `Function, Map>` | -| `filter()` | `Closure` with `tags.x == 'y'` | `SampleFilter` | `Predicate>` | -| `forEach()` | `Closure` with `(prefix, tags) -> ...` | `ForEachFunction` | `BiConsumer>` | -| `decorate()` | `Closure` with `entity -> ...` | `DecorateFunction` | `Consumer` | -| `instance(..., closure)` | `Closure` with `tags -> Map.of(...)` | `PropertiesExtractor` | `Function, Map>` | - -Source location in upstream: `oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java` - -### 3.3 MAL Transpiler: AST Mapping Rules - -The `MalToJavaTranspiler` (~1,230 lines) parses each MAL expression string into a Groovy AST at `Phases.CONVERSION` (no code execution), then walks the AST to emit equivalent Java code. - -#### Expression Mappings - -| Groovy Construct | Java Output | Notes | -|-----------------|-------------|-------| -| `metric_name` (bare property) | `samples.getOrDefault("metric_name", SampleFamily.EMPTY)` | Replaces `propertyMissing()` dispatch | -| `.sum(['a','b'])` | `.sum(List.of("a", "b"))` | Direct method call | -| `.tagEqual('resource', 'cpu')` | `.tagEqual("resource", "cpu")` | Direct method call | -| `100 * metric` | `metric.multiply(100)` | Commutative: operands swapped | -| `100 - metric` | `metric.minus(100).negative()` | Non-commutative: negate | -| `100 / metric` | `metric.newValue(v -> 100 / v)` | Non-commutative: newValue | -| `metricA / metricB` | `metricA.div(metricB)` | SampleFamily-SampleFamily op | -| `.tag({tags -> tags.cluster = ...})` | `.tag(tags -> { tags.put("cluster", ...); return tags; })` | Closure -> lambda | -| `.filter({tags -> tags.job_name in [...]})` | `.filter(tags -> "...".equals(tags.get("job_name")))` | Closure -> predicate | -| `.forEach(['a','b'], {p, tags -> ...})` | `.forEach(List.of("a","b"), (p, tags) -> { ... })` | Closure -> BiConsumer | -| `.decorate({entity -> ...})` | `.decorate(entity -> { ... })` | Closure -> Consumer | -| `.instance(..., {tags -> Map.of(...)})` | `.instance(..., tags -> Map.of(...))` | Closure -> Function | -| `Layer.K8S` | `Layer.K8S` | Enum constant, passed through | -| `time()` | `Instant.now().getEpochSecond()` | Direct Java API | -| `AVG`, `SUM`, etc. | `DownsamplingType.AVG`, etc. | Enum constant reference | - -#### Closure Body Translation - -Inside closures, Groovy property-style access is mapped to explicit `Map` operations: - -| Groovy Closure Pattern | Java Lambda Output | -|----------------------|-------------------| -| `tags.key = "value"` | `tags.put("key", "value")` | -| `tags.key` (read) | `tags.get("key")` | -| `tags.remove("key")` | `tags.remove("key")` | -| `tags.key == "value"` | `"value".equals(tags.get("key"))` | -| `tags.key != "value"` | `!"value".equals(tags.get("key"))` | -| `tags.key in ["a","b"]` | `List.of("a","b").contains(tags.get("key"))` | -| `if/else` in closure | `if/else` in lambda | -| `entity.serviceId = val` | `entity.setServiceId(val)` | - -#### Filter Expression Mappings - -Filter expressions (`filter: "{ tags -> ... }"`) generate `MalFilter` implementations: - -| Groovy Filter | Java Output | -|--------------|-------------| -| `tags.job_name == 'mysql'` | `"mysql".equals(tags.get("job_name"))` | -| `tags.job_name != 'test'` | `!"test".equals(tags.get("job_name"))` | -| `tags.job_name in ['a','b']` | `List.of("a","b").contains(tags.get("job_name"))` | -| `cond1 && cond2` | `cond1 && cond2` | -| `cond1 \|\| cond2` | `cond1 \|\| cond2` | -| `!cond` | `!cond` | -| `tags.job_name` (truthiness) | `tags.get("job_name") != null` | - -### 3.4 LAL Transpiler: AST Mapping Rules - -The `LalToJavaTranspiler` (~950 lines) handles LAL's `@CompileStatic` delegation-based DSL. LAL scripts have a fundamentally different structure from MAL -- they are statement-based builder patterns rather than expression-based computations. - -#### Statement Mappings - -| Groovy Construct | Java Output | -|-----------------|-------------| -| `filter { ... }` | Body unwrapped, emitted directly on `filterSpec` | -| `json {}` | `filterSpec.json()` | -| `json { abortOnFailure false }` | `filterSpec.json(jp -> { jp.abortOnFailure(false); })` | -| `text { regexp /pattern/ }` | `filterSpec.text(tp -> { tp.regexp("pattern"); })` | -| `yaml {}` | `filterSpec.yaml()` | -| `extractor { ... }` | `filterSpec.extractor(ext -> { ... })` | -| `sink { ... }` | `filterSpec.sink(s -> { ... })` | -| `abort {}` | `filterSpec.abort()` | -| `service parsed.service as String` | `ext.service(String.valueOf(getAt(binding.parsed(), "service")))` | -| `layer parsed.layer as String` | `ext.layer(String.valueOf(getAt(binding.parsed(), "layer")))` | -| `tag(key: val)` | `ext.tag(Map.of("key", val))` | -| `timestamp parsed.time as String` | `ext.timestamp(String.valueOf(getAt(binding.parsed(), "time")))` | - -#### Property Access and Safe Navigation - -| Groovy Pattern | Java Output | -|---------------|-------------| -| `parsed.field` | `getAt(binding.parsed(), "field")` | -| `parsed.field.nested` | `getAt(getAt(binding.parsed(), "field"), "nested")` | -| `parsed?.field?.nested` | `((__v0 = binding.parsed()) == null ? null : ((__v1 = getAt(__v0, "field")) == null ? null : getAt(__v1, "nested")))` | -| `log.tags` | `binding.log().getTags()` | - -#### Cast and Type Handling - -| Groovy Pattern | Java Output | -|---------------|-------------| -| `expr as String` | `String.valueOf(expr)` | -| `expr as Long` | `toLong(expr)` | -| `expr as Integer` | `toInt(expr)` | -| `expr as Boolean` | `toBoolean(expr)` | -| `"${expr}"` (GString) | `"" + expr` | - -#### LAL Spec Consumer Overloads - -LAL spec classes (`FilterSpec`, `ExtractorSpec`, `SinkSpec`) get additional method overloads accepting `java.util.function.Consumer` alongside existing Groovy `Closure` parameters: - -```java -// FilterSpec - existing -public void extractor(Closure cl) { ... } -// FilterSpec - new overload -public void extractor(Consumer consumer) { ... } - -// SinkSpec - existing -public void sampler(Closure cl) { ... } -// SinkSpec - new overload -public void sampler(Consumer consumer) { ... } -``` - -Methods requiring Consumer overloads: `text()`, `json()`, `yaml()`, `extractor()`, `sink()`, `slowSql()`, `sampledTrace()`, `metrics()`, `sampler()`, `enforcer()`, `dropper()`. - -#### SHA-256 Deduplication - -LAL manifest is keyed by SHA-256 hash of the DSL content. Identical scripts across different YAML files share one compiled class. In practice, 10 LAL rules map to 6 unique classes. - -### 3.5 Hierarchy Script: v1/v2 Module Split - -The hierarchy matching rules in `hierarchy-definition.yml` use `GroovyShell.evaluate()` to compile 4 Groovy closures at runtime. Unlike MAL/LAL, hierarchy does not need a transpiler (only 4 rules, finite set), but it follows the same v1/v2/checker module pattern for consistency and to remove Groovy from `server-core`. - -#### Current State (server-core, Groovy-coupled) - -`HierarchyDefinitionService.java` lives in `server-core` and is registered as a `Service` in `CoreModule`. Its inner class `MatchingRule` holds a Groovy `Closure`: - -```java -// server-core/...config/HierarchyDefinitionService.java (current) -public static class MatchingRule { - private final String name; - private final String expression; - private final Closure closure; // groovy.lang.Closure - - public MatchingRule(final String name, final String expression) { - GroovyShell sh = new GroovyShell(); - closure = (Closure) sh.evaluate(expression); // Groovy at runtime - } -} -``` - -This `MatchingRule` is referenced by three classes in `server-core`: -- `HierarchyDefinitionService` -- builds the rule map from YAML -- `HierarchyService` -- calls `matchingRule.getClosure().call(service, comparedService)` for auto-matching -- `HierarchyQueryService` -- reads the hierarchy definition map - -#### Step 1: Make server-core Groovy-Free - -Refactor `MatchingRule` in `server-core` to use a Java functional interface instead of `Closure`: - -```java -// server-core/...config/HierarchyDefinitionService.java (refactored) -public static class MatchingRule { - private final String name; - private final String expression; - private final BiFunction matcher; // pure Java - - public MatchingRule(final String name, final String expression, - final BiFunction matcher) { - this.name = name; - this.expression = expression; - this.matcher = matcher; - } - - public boolean match(Service upper, Service lower) { - return matcher.apply(upper, lower); - } -} -``` - -`HierarchyDefinitionService.init()` no longer compiles Groovy expressions itself. Instead, it receives a `Map>` (the rule registry) from outside -- injected by whichever implementation module (v1 or v2) is active. - -`HierarchyService` changes from `matchingRule.getClosure().call(u, l)` to `matchingRule.match(u, l)`. - -Remove all `groovy.lang.*` imports from `server-core`. - -#### Step 2: hierarchy-v1 (Groovy-based, for checker only) - -```java -// analyzer/hierarchy-v1/.../GroovyHierarchyRuleProvider.java -public class GroovyHierarchyRuleProvider { - public static Map> buildRules( - Map ruleExpressions) { - Map> rules = new HashMap<>(); - GroovyShell sh = new GroovyShell(); - ruleExpressions.forEach((name, expression) -> { - Closure closure = (Closure) sh.evaluate(expression); - rules.put(name, (u, l) -> closure.call(u, l)); - }); - return rules; - } -} -``` - -This module depends on Groovy and wraps the original `GroovyShell.evaluate()` logic. It is NOT included in the runtime classpath -- only used by the checker. - -#### Step 3: hierarchy-v2 (Pure Java, for runtime) - -```java -// analyzer/hierarchy-v2/.../JavaHierarchyRuleProvider.java -public class JavaHierarchyRuleProvider { - private static final Map> RULE_REGISTRY; - static { - RULE_REGISTRY = new HashMap<>(); - RULE_REGISTRY.put("name", - (u, l) -> Objects.equals(u.getName(), l.getName())); - RULE_REGISTRY.put("short-name", - (u, l) -> Objects.equals(u.getShortName(), l.getShortName())); - RULE_REGISTRY.put("lower-short-name-remove-ns", (u, l) -> { - String sn = l.getShortName(); - int dot = sn.lastIndexOf('.'); - return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); - }); - RULE_REGISTRY.put("lower-short-name-with-fqdn", (u, l) -> { - String sn = u.getShortName(); - int colon = sn.lastIndexOf(':'); - return colon > 0 && Objects.equals( - sn.substring(0, colon), - l.getShortName() + ".svc.cluster.local"); - }); - } - - public static Map> buildRules( - Map ruleExpressions) { - Map> rules = new HashMap<>(); - ruleExpressions.forEach((name, expression) -> { - BiFunction fn = RULE_REGISTRY.get(name); - if (fn == null) { - throw new IllegalArgumentException( - "Unknown hierarchy matching rule: " + name - + ". Known rules: " + RULE_REGISTRY.keySet()); - } - rules.put(name, fn); - }); - return rules; - } -} -``` - -Unknown rule names fail fast at startup with `IllegalArgumentException`. The YAML file (`hierarchy-definition.yml`) continues to reference rule names (`name`, `short-name`, etc.) -- the Groovy expression strings in `auto-matching-rules` become documentation-only at runtime. - -#### Step 4: hierarchy-v1-v2-checker - -```java -// analyzer/hierarchy-v1-v2-checker/.../HierarchyRuleComparisonTest.java -class HierarchyRuleComparisonTest { - // Load rule expressions from hierarchy-definition.yml - // For each rule: - // Path A: GroovyHierarchyRuleProvider.buildRules() (v1) - // Path B: JavaHierarchyRuleProvider.buildRules() (v2) - // Construct test Service pairs (matching and non-matching cases) - // Assert v1.match(u, l) == v2.match(u, l) for all test pairs -} -``` - -Test cases cover all 4 rules with realistic service name patterns: -- `name`: exact match and mismatch -- `short-name`: exact shortName match and mismatch -- `lower-short-name-remove-ns`: `"svc" == "svc.namespace"` and edge cases (no dot, empty) -- `lower-short-name-with-fqdn`: `"db:3306"` vs `"db.svc.cluster.local"` and edge cases (no colon, wrong suffix) - ---- - -## 4. Module Structure - -### 4.1 Upstream Module Layout - -``` -oap-server/ - server-core/ # MODIFIED: MatchingRule uses BiFunction (no Groovy imports) - - analyzer/ - meter-analyzer/ # Modified: add MalExpression, functional interfaces - log-analyzer/ # Modified: add LalExpression, Consumer overloads - - mal-lal-v1/ # NEW: Move existing Groovy-based code here - meter-analyzer-v1/ # Original MAL (GroovyShell + ExpandoMetaClass) - log-analyzer-v1/ # Original LAL (GroovyShell + @CompileStatic) - - mal-lal-v2/ # NEW: Pure Java transpiler-based implementations - meter-analyzer-v2/ # MalExpression loader + functional interface dispatch - log-analyzer-v2/ # LalExpression loader + Consumer dispatch - mal-transpiler/ # Build-time: Groovy AST -> Java source (MAL) - lal-transpiler/ # Build-time: Groovy AST -> Java source (LAL) - - mal-lal-v1-v2-checker/ # NEW: Dual-path comparison tests (MAL + LAL) - 73 MAL test classes (1,281 assertions) - 5 LAL test classes (19 assertions) - - hierarchy-v1/ # NEW: Groovy-based hierarchy rule provider (checker only) - hierarchy-v2/ # NEW: Pure Java hierarchy rule provider (runtime) - hierarchy-v1-v2-checker/ # NEW: Dual-path comparison tests (hierarchy) -``` - -### 4.2 Dependency Graph - -``` -mal-transpiler ──────────────> groovy (build-time only, for AST parsing) -lal-transpiler ──────────────> groovy (build-time only, for AST parsing) -hierarchy-v1 ────────────────> groovy (checker only, not runtime) - -meter-analyzer-v2 ──────────> meter-analyzer (interfaces + SampleFamily) -log-analyzer-v2 ────────────> log-analyzer (interfaces + spec classes) -hierarchy-v2 ───────────────> server-core (MatchingRule with BiFunction) - -mal-lal-v1-v2-checker ──────> mal-lal-v1 (Groovy path) -mal-lal-v1-v2-checker ──────> mal-lal-v2 (Java path) -hierarchy-v1-v2-checker ────> hierarchy-v1 (Groovy path) -hierarchy-v1-v2-checker ────> hierarchy-v2 (Java path) - -server-starter ─────────────> meter-analyzer-v2 (runtime, no Groovy) -server-starter ─────────────> log-analyzer-v2 (runtime, no Groovy) -server-starter ─────────────> hierarchy-v2 (runtime, no Groovy) -server-starter ────────────X─> mal-lal-v1 (NOT in runtime) -server-starter ────────────X─> hierarchy-v1 (NOT in runtime) -``` - -### 4.3 Key Design Principle: No Coexistence - -v1 (Groovy) and v2 (Java) never coexist in the OAP runtime classpath. The `mal-lal-v1` and `hierarchy-v1` modules are only dependencies of their respective checker modules for CI validation. The runtime (`server-starter`) depends only on v2 modules. - ---- - -## 5. Implementation Steps - -### Phase 1: Interfaces and SampleFamily Modifications - -**Files to modify:** - -1. **Create `MalExpression.java`** in `meter-analyzer` - - Path: `oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java` - -2. **Create `MalFilter.java`** in `meter-analyzer` - - Path: `oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java` - -3. **Create functional interfaces** in `meter-analyzer` - - `TagFunction extends Function, Map>` - - `SampleFilter extends Predicate>` - - `ForEachFunction extends BiConsumer>` - - `DecorateFunction extends Consumer` - - `PropertiesExtractor extends Function, Map>` - -4. **Add overloads to `SampleFamily.java`** - - Add `tag(TagFunction)` alongside existing `tag(Closure)` - - Add `filter(SampleFilter)` alongside existing `filter(Closure)` - - Add `forEach(List, ForEachFunction)` alongside existing `forEach(List, Closure)` - - Add `decorate(DecorateFunction)` alongside existing `decorate(Closure)` - - Add `instance(..., PropertiesExtractor)` alongside existing `instance(..., Closure)` - -5. **Create `LalExpression.java`** in `log-analyzer` - - Path: `oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java` - -6. **Add Consumer overloads to LAL spec classes** - - `FilterSpec`: `text(Consumer)`, `json(Consumer)`, `yaml(Consumer)`, `extractor(Consumer)`, `sink(Consumer)`, `filter(Consumer)` - - `ExtractorSpec`: `slowSql(Consumer)`, `sampledTrace(Consumer)`, `metrics(Consumer)` - - `SinkSpec`: `sampler(Consumer)`, `enforcer(Consumer)`, `dropper(Consumer)` - -### Phase 2: MAL Transpiler - -**New module: `oap-server/analyzer/mal-lal-v2/mal-transpiler/`** - -1. **`MalToJavaTranspiler.java`** (~1,230 lines) - - Uses `org.codehaus.groovy.control.CompilationUnit` at `Phases.CONVERSION` - - Walks AST via recursive visitor pattern - - Core methods: - - `transpileExpression(String className, String expression)` -> generates Java source for `MalExpression` - - `transpileFilter(String className, String filterLiteral)` -> generates Java source for `MalFilter` - - `collectSampleNames(Expression expr)` -> extracts all metric name references - - `visitExpression(Expression node)` -> recursive Java code emitter - - `visitClosureExpression(ClosureExpression node, String contextType)` -> closure-to-lambda - - `compileAll()` -> batch `javac` compile + manifest generation - -2. **Maven integration**: `exec-maven-plugin` during `generate-sources` phase - - Reads all MAL YAML files from resources - - Generates Java source to `target/generated-sources/mal/` - - Compiles to `target/classes/` - - Writes `META-INF/mal-expressions.txt` and `META-INF/mal-filter-expressions.properties` - -### Phase 3: LAL Transpiler - -**New module: `oap-server/analyzer/mal-lal-v2/lal-transpiler/`** - -1. **`LalToJavaTranspiler.java`** (~950 lines) - - Same AST approach as MAL but statement-based emission - - Core methods: - - `transpile(String className, String dslText)` -> generates Java source for `LalExpression` - - `emitStatement(Statement node, String receiver, BindingContext ctx)` -> statement emitter - - `visitConditionExpr(Expression node)` -> boolean expression emitter - - `emitPropertyAccess(PropertyExpression node)` -> `getAt()` with null safety - - SHA-256 deduplication: identical DSL content shares one class - - Helper methods in generated class: `getAt()`, `toLong()`, `toInt()`, `toBoolean()`, `isTruthy()`, `isNonEmptyString()` - -2. **Maven integration**: same `exec-maven-plugin` approach - - Writes `META-INF/lal-expressions.txt` (SHA-256 hash -> FQCN) - -### Phase 4: Runtime Loading (v2 Modules) - -**New module: `oap-server/analyzer/mal-lal-v2/meter-analyzer-v2/`** - -1. **Modified `DSL.java`** (MAL runtime): - ```java - public static Expression parse(String metricName, String expression) { - Map manifest = loadManifest("META-INF/mal-expressions.txt"); - String className = manifest.get(metricName); - MalExpression malExpr = (MalExpression) Class.forName(className) - .getDeclaredConstructor().newInstance(); - return new Expression(metricName, expression, malExpr); - } - ``` - -2. **Modified `Expression.java`** (MAL runtime): - - Wraps `MalExpression` instead of `DelegatingScript` - - `run()` calls `malExpression.run(sampleFamilies)` directly - - No `ExpandoMetaClass`, no `ExpressionDelegate`, no `ThreadLocal` - -3. **Modified `FilterExpression.java`** (MAL runtime): - - Loads `MalFilter` from `META-INF/mal-filter-expressions.properties` - - `filter()` calls `malFilter::test` via `SampleFamily.filter(SampleFilter)` - -**New module: `oap-server/analyzer/mal-lal-v2/log-analyzer-v2/`** - -4. **Modified `DSL.java`** (LAL runtime): - - Computes SHA-256 of DSL text, loads class from `META-INF/lal-expressions.txt` - - `evaluate()` calls `lalExpression.execute(filterSpec, binding)` directly - - No `GroovyShell`, no `LALDelegatingScript` - -### Phase 5: Hierarchy v1/v2 Module Split - -**Step 5a: Refactor `server-core` to remove Groovy** - -**File to modify:** `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/config/HierarchyDefinitionService.java` - -1. Change `MatchingRule.closure` from `Closure` to `BiFunction matcher` -2. Add constructor that accepts the `BiFunction` matcher directly -3. Replace `getClosure()` with `match(Service upper, Service lower)` method -4. Change `init()` to accept a rule registry (`Map>`) from outside instead of calling `GroovyShell.evaluate()` internally -5. Remove all `groovy.lang.*` imports - -**File to modify:** `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/hierarchy/HierarchyService.java` - -6. Change `matchingRule.getClosure().call(service, comparedService)` (lines 201-203, 220-222) to `matchingRule.match(service, comparedService)` - -**Step 5b: Create hierarchy-v1 module** - -**New module:** `oap-server/analyzer/hierarchy-v1/` - -1. `GroovyHierarchyRuleProvider.java`: wraps original `GroovyShell.evaluate()` logic -2. Takes `Map` (rule name -> Groovy expression) from YAML -3. Returns `Map>` by evaluating closures -4. Depends on Groovy -- NOT included in runtime, only used by checker - -**Step 5c: Create hierarchy-v2 module** - -**New module:** `oap-server/analyzer/hierarchy-v2/` - -1. `JavaHierarchyRuleProvider.java`: static `RULE_REGISTRY` with 4 Java lambdas -2. Takes `Map` (rule name -> Groovy expression) from YAML (expression ignored, only name used for registry lookup) -3. Returns `Map>` from the registry -4. Fails fast with `IllegalArgumentException` for unknown rule names -5. Zero Groovy dependency - -### Phase 6: Comparison Test Suites - -**New module: `oap-server/analyzer/mal-lal-v1-v2-checker/`** - -1. **MAL comparison tests** (73 test classes): - - Base class `MALScriptComparisonBase` runs dual-path comparison: - - Path A: Fresh Groovy compilation with upstream `CompilerConfiguration` - - Path B: Load transpiled `MalExpression` from manifest - - Both receive identical `Map` input - - Compare: `ExpressionParsingContext` (scope, function, datatype, downsampling), `SampleFamily` result (values, labels, entity descriptions) - - JUnit 5 `@TestFactory` generates `DynamicTest` per metric rule - - Test data must be non-trivial to prevent vacuous agreement (both returning empty/null) - -2. **LAL comparison tests** (5 test classes): - - Base class `LALScriptComparisonBase` runs dual-path comparison: - - Path A: Groovy with `@CompileStatic` + `LALPrecompiledExtension` - - Path B: Load `LalExpression` from manifest via SHA-256 - - Compare: `shouldAbort()`, `shouldSave()`, LogData.Builder state, metrics container, `databaseSlowStatement`, `sampledTraceBuilder` - -3. **Test statistics**: 1,281 MAL assertions + 19 LAL assertions = 1,300 total - -**New module: `oap-server/analyzer/hierarchy-v1-v2-checker/`** - -4. **Hierarchy comparison tests**: - - Load rule expressions from `hierarchy-definition.yml` - - For each of the 4 rules: - - Path A: `GroovyHierarchyRuleProvider.buildRules()` (v1, Groovy closures) - - Path B: `JavaHierarchyRuleProvider.buildRules()` (v2, Java lambdas) - - Construct test `Service` pairs covering matching and non-matching cases: - - `name`: exact match `("svc", "svc")` -> true, `("svc", "other")` -> false - - `short-name`: shortName match/mismatch - - `lower-short-name-remove-ns`: `"svc"` vs `"svc.namespace"` -> true, no dot -> false, empty -> false - - `lower-short-name-with-fqdn`: `"db:3306"` vs `"db.svc.cluster.local"` -> true, no colon -> false, wrong suffix -> false - - Assert `v1.match(u, l) == v2.match(u, l)` for all test pairs - -### Phase 7: Cleanup and Dependency Removal - -1. **Move v1 code to `mal-lal-v1/` and `hierarchy-v1/`** (or mark as `test`) -2. **Remove Groovy from runtime classpath**: `groovy-5.0.3.jar` (~7 MB) becomes test-only -3. **Remove from `server-starter` dependencies**: replace v1 with v2 module references for MAL, LAL, and hierarchy -4. **Remove `NumberClosure.java`**: no longer needed without `ExpandoMetaClass` -5. **Remove `ExpressionDelegate.propertyMissing()`**: replaced by `samples.getOrDefault()` -6. **Remove Groovy closure overloads from `SampleFamily`** (after v1 is fully deprecated) -7. **Remove `LALDelegatingScript.java`**: replaced by `LalExpression` interface -8. **Verify `server-core` has zero Groovy imports**: `HierarchyDefinitionService` and `HierarchyService` now use `BiFunction` only - -### Phase 8: Replace v2 Manifest Loading with Real Compilers (ANTLR4 + Javassist) - -Phase 7 completed: v2 modules are standalone (zero Groovy), v1 depends on v2. Currently, v2 loads transpiled classes via manifest files (`META-INF/mal-expressions.txt`, `META-INF/lal-expressions.txt`) that were pre-compiled at build time. This prevents on-demand config changes since MAL/LAL/hierarchy configs are in the final package and users may want to modify them. - -The goal is to replace this manifest-based approach with **real compilers** following the OAL pattern: ANTLR4 grammar -> parser -> model -> Javassist class generation -> listener notification. This enables runtime compilation when configs change. - -#### Module Renaming: Drop `-v2` Suffix - -Since v1 modules move to `test/script-compiler/`, the v2 modules become the primary ones and lose the `-v2` suffix: -- `meter-analyzer-v2` -> `meter-analyzer` (package stays `o.a.s.oap.meter.analyzer`) -- `log-analyzer-v2` -> `log-analyzer` (package stays `o.a.s.oap.log.analyzer`) -- `hierarchy-v2` -> `hierarchy` (no v1 name conflict since v1 moves out) -- `.v2.` sub-packages (`dsl.v2.DSL`, `dsl.v2.Binding`, etc.) merge back into parent packages (`dsl.DSL`, `dsl.Binding`) - -#### Target Module Structure - -``` -oap-server/ - mal-grammar/ NEW — ANTLR4 grammar for MAL expressions - lal-grammar/ NEW — ANTLR4 grammar for LAL scripts - hierarchy-rule-grammar/ NEW — ANTLR4 grammar for hierarchy matching rules - -oap-server/analyzer/ - agent-analyzer/ (stays) - event-analyzer/ (stays) - meter-analyzer/ (renamed from meter-analyzer-v2, runtime MAL, calls mal-compiler) - log-analyzer/ (renamed from log-analyzer-v2, runtime LAL, calls lal-compiler) - hierarchy/ (renamed from hierarchy-v2, calls hierarchy-rule-compiler) - mal-compiler/ NEW — MAL expression compiler engine - lal-compiler/ NEW — LAL script compiler engine - hierarchy-rule-compiler/ NEW — hierarchy rule compiler engine - -test/script-compiler/ NEW — aggregator for v1/transpiler/checker (not in dist) - mal-groovy/ <- meter-analyzer v1 (Groovy) - lal-groovy/ <- log-analyzer v1 (Groovy) - hierarchy-groovy/ <- hierarchy-v1 (Groovy) - mal-transpiler/ <- mal-transpiler - lal-transpiler/ <- lal-transpiler - mal-lal-v1-v2-checker/ <- mal-lal-v1-v2-checker - hierarchy-v1-v2-checker/ <- hierarchy-v1-v2-checker -``` - -#### Generated Class Grouping by Config File Name - -MAL metrics come from YAML config files (e.g., `oap.yaml`, `spring-micrometer.yaml`). Each file contains multiple `metricsRules`. The compiler groups generated classes by source file name. - -- **MAL Compiler API**: `MALCompilerEngine.compile(configFileName, MetricRuleConfig)` -> grouped by file -- **Generated class naming**: `rt..MalExpr_`, e.g., `rt.oap.MalExpr_instance_jvm_cpu` -- **LAL Compiler API**: `LALCompilerEngine.compile(configFileName, List)` -> grouped by file -- **Generated class naming**: `rt..LalExpr_` - -#### Eliminate `ExpressionParsingContext` ThreadLocal (MAL only) - -The current MAL `run()` method serves dual purposes controlled by a ThreadLocal: -1. **Parse phase** (startup): `ExpressionParsingContext` ThreadLocal is set, `run()` is called with an empty map to discover which metric names the expression references -2. **Runtime phase** (every ingestion cycle): ThreadLocal is not set, `run()` computes the actual result - -This is eliminated by extracting metadata statically. The `MalExpression` interface gains a `metadata()` method: - -```java -public interface MalExpression { - /** Pure computation. No side effects. */ - SampleFamily run(Map samples); - - /** Compile-time metadata -- sample names, scope, downsampling, etc. */ - ExpressionMetadata metadata(); -} -``` - -The ANTLR4 compiler extracts all metadata from the parse tree at compile time: - -| Metadata | Extracted from | -|--|--| -| `sampleNames` | Bare identifiers (metric references) | -| `scopeType` | Terminal method: `.service()`, `.instance()`, `.endpoint()` | -| `downsampling` | Aggregation arg: `AVG`, `SUM`, `MAX`, etc. | -| `percentiles` | `.percentile()` call arguments | -| `isHistogram` | Presence of `.histogram()` in chain | - -Generated class emits metadata as static fields: - -```java -public class MalExpr_instance_jvm_cpu implements MalExpression { - private static final ExpressionMetadata METADATA = new ExpressionMetadata( - List.of("instance_jvm_cpu"), // sampleNames - ScopeType.SERVICE_INSTANCE, // from .service()/instance() call - DownsamplingType.AVG // from aggregation - ); - - @Override - public ExpressionMetadata metadata() { return METADATA; } - - @Override - public SampleFamily run(Map samples) { - return ((SampleFamily) samples.getOrDefault("instance_jvm_cpu", SampleFamily.EMPTY)) - .sum(List.of("service", "instance")); - } -} -``` - -Result: `run()` is pure computation, `metadata()` is static facts, `ExpressionParsingContext` and its ThreadLocal are deleted. No dry run with empty map at startup. - -LAL and hierarchy do **not** have this problem -- LAL passes `Binding` explicitly as a parameter, hierarchy rules are stateless lambdas. - -#### Implementation Sub-Phases - -- 8.1: Rename modules (drop -v2 suffix, flatten .v2. sub-packages) -- 8.2: Grammar modules (ANTLR4 .g4 files) -- 8.3: Compiler model + parser (no code gen) -- 8.4: Javassist code generation (including static `ExpressionMetadata` on generated MAL classes) -- 8.5: Engine integration (wire compilers into renamed modules, delete `ExpressionParsingContext`) -- 8.6: Move v1/transpiler/checker to test/script-compiler/ -- 8.7: Cleanup (remove manifests, verify zero Groovy) - ---- - -## 6. What Gets Removed from Runtime - -| Component | Current | After | -|-----------|---------|-------| -| `GroovyShell.parse()` in MAL `DSL.java` | 1,250+ calls at boot | `Class.forName()` from manifest | -| `GroovyShell.evaluate()` in MAL `FilterExpression.java` | 29 filter compilations | `Class.forName()` from manifest | -| `GroovyShell.parse()` in LAL `DSL.java` | 10 script compilations | `Class.forName()` from manifest | -| `GroovyShell.evaluate()` in `HierarchyDefinitionService` | 4 rule compilations | `hierarchy-v2` Java lambda registry | -| `Closure` in `MatchingRule` | Groovy closure in `server-core` | `BiFunction` (Groovy-free `server-core`) | -| `ExpandoMetaClass` registration in `Expression.empower()` | Runtime metaclass on `Number` | Direct `multiply()`/`div()` method calls | -| `ExpressionDelegate.propertyMissing()` | Dynamic property dispatch | `samples.getOrDefault()` | -| `groovy.lang.Closure` in `SampleFamily` | 5 method signatures | Java functional interfaces | -| `groovy-5.0.3.jar` runtime dependency | ~7 MB on classpath | Removed (build-time only) | - ---- - -## 7. Transpiler Technical Details - -### 7.1 AST Parsing Strategy - -Both transpilers use Groovy's `CompilationUnit` at `Phases.CONVERSION`: - -```java -CompilationUnit cu = new CompilationUnit(); -cu.addSource("expression", new StringReaderSource( - new StringReader(groovyCode), cu.getConfiguration())); -cu.compile(Phases.CONVERSION); // Parse + AST transform, no codegen -ModuleNode ast = cu.getAST(); -``` - -This extracts the complete syntax tree without: -- Generating Groovy bytecode -- Resolving classes on classpath -- Activating MOP or MetaClass - -The Groovy dependency is therefore **build-time only**. - -### 7.2 MAL Arithmetic Operand Swap - -The transpiler must replicate the exact behavior of upstream's `ExpandoMetaClass` on `Number`. When a `Number` appears on the left side of an operator with a `SampleFamily` on the right, the operands must be handled carefully: - -``` -N + SF -> SF.plus(N) // commutative, swap operands -N - SF -> SF.minus(N).negative() // non-commutative: (N - SF) = -(SF - N) -N * SF -> SF.multiply(N) // commutative, swap operands -N / SF -> SF.newValue(v -> N / v) // non-commutative: per-sample (N / sample_value) -``` - -The transpiler detects `Number` vs `SampleFamily` operand types by tracking whether a sub-expression references sample names (metric properties) or is a numeric literal/constant. - -### 7.3 Batch Compilation - -Generated Java sources are compiled in a single `javac` invocation via `javax.tools.JavaCompiler`: - -```java -JavaCompiler compiler = ToolProvider.getSystemJavaCompiler(); -List sources = /* all generated .java files */; -compiler.getTask(null, fileManager, diagnostics, options, null, sources).call(); -``` - -This avoids 1,250+ individual `javac` invocations and provides full cross-file type checking. - -### 7.4 Manifest Format - -**`META-INF/mal-expressions.txt`**: -``` -metric_name_1=org.apache.skywalking.oap.meter.analyzer.dsl.generated.MalExpr_metric_name_1 -metric_name_2=org.apache.skywalking.oap.meter.analyzer.dsl.generated.MalExpr_metric_name_2 -... -``` - -**`META-INF/mal-filter-expressions.properties`**: -``` -{ tags -> tags.job_name == 'mysql-monitoring' }=org.apache.skywalking.oap.meter.analyzer.dsl.generated.MalFilter_0a1b2c3d -... -``` - -**`META-INF/lal-expressions.txt`**: -``` -sha256_hash_1=org.apache.skywalking.oap.log.analyzer.dsl.generated.LalExpr_sha256_hash_1 -sha256_hash_2=org.apache.skywalking.oap.log.analyzer.dsl.generated.LalExpr_sha256_hash_2 -... -``` - ---- - -## 8. Worked Examples - -### 8.1 MAL Expression: K8s Node CPU Capacity - -**Groovy (upstream):** -```groovy -kube_node_status_capacity.tagEqual('resource', 'cpu') - .sum(['node']) - .tag({tags -> tags.node_name = tags.node; tags.remove("node")}) - .service(['node_name'], Layer.K8S) -``` - -**Transpiled Java:** -```java -public class MalExpr_k8s_node_cpu implements MalExpression { - @Override - public SampleFamily run(Map samples) { - return samples.getOrDefault("kube_node_status_capacity", SampleFamily.EMPTY) - .tagEqual("resource", "cpu") - .sum(List.of("node")) - .tag(tags -> { - tags.put("node_name", tags.get("node")); - tags.remove("node"); - return tags; - }) - .service(List.of("node_name"), Layer.K8S); - } -} -``` - -### 8.2 MAL Expression: Arithmetic with Number on Left - -**Groovy (upstream):** -```groovy -100 - container_cpu_usage / container_resource_limit_cpu * 100 -``` - -**Transpiled Java:** -```java -public class MalExpr_cpu_percent implements MalExpression { - @Override - public SampleFamily run(Map samples) { - return samples.getOrDefault("container_cpu_usage", SampleFamily.EMPTY) - .div(samples.getOrDefault("container_resource_limit_cpu", SampleFamily.EMPTY)) - .multiply(100) - .minus(100) - .negative(); - } -} -``` - -### 8.3 MAL Filter Expression - -**Groovy (upstream):** -```groovy -{ tags -> tags.job_name == 'mysql-monitoring' } -``` - -**Transpiled Java:** -```java -public class MalFilter_mysql implements MalFilter { - @Override - public boolean test(Map tags) { - return "mysql-monitoring".equals(tags.get("job_name")); - } -} -``` - -### 8.4 LAL Expression: MySQL Slow SQL - -**Groovy (upstream):** -```groovy -filter { - json {} - extractor { - layer parsed.layer as String - service parsed.service as String - timestamp parsed.time as String - if (tag("LOG_KIND") == "SLOW_SQL") { - slowSql { - id parsed.id as String - statement parsed.statement as String - latency parsed.query_time as Long - } - } - } - sink {} -} -``` - -**Transpiled Java:** -```java -public class LalExpr_mysql_slowsql implements LalExpression { - - @Override - public void execute(FilterSpec filterSpec, Binding binding) { - filterSpec.json(); - filterSpec.extractor(ext -> { - ext.layer(String.valueOf(getAt(binding.parsed(), "layer"))); - ext.service(String.valueOf(getAt(binding.parsed(), "service"))); - ext.timestamp(String.valueOf(getAt(binding.parsed(), "time"))); - if ("SLOW_SQL".equals(ext.tag("LOG_KIND"))) { - ext.slowSql(ss -> { - ss.id(String.valueOf(getAt(binding.parsed(), "id"))); - ss.statement(String.valueOf(getAt(binding.parsed(), "statement"))); - ss.latency(toLong(getAt(binding.parsed(), "query_time"))); - }); - } - }); - filterSpec.sink(s -> {}); - } - - private static Object getAt(Object obj, String key) { - if (obj instanceof Binding.Parsed) return ((Binding.Parsed) obj).getAt(key); - if (obj instanceof Map) return ((Map) obj).get(key); - return null; - } - - private static long toLong(Object val) { - if (val instanceof Number) return ((Number) val).longValue(); - if (val instanceof String) return Long.parseLong((String) val); - return 0L; - } -} -``` - -### 8.5 Hierarchy Rule: lower-short-name-remove-ns - -**Groovy (upstream, in hierarchy-definition.yml):** -```groovy -{ (u, l) -> { - if(l.shortName.lastIndexOf('.') > 0) - return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); - return false; -} } -``` - -**Java replacement (in HierarchyDefinitionService.java):** -```java -RULE_REGISTRY.put("lower-short-name-remove-ns", (u, l) -> { - String sn = l.getShortName(); - int dot = sn.lastIndexOf('.'); - return dot > 0 && Objects.equals(u.getShortName(), sn.substring(0, dot)); -}); -``` - ---- - -## 9. Verification Strategy - -### 9.1 Dual-Path Comparison Testing - -Every generated Java class is validated against the original Groovy behavior in CI: - -``` -For each MAL YAML file: - For each metric rule: - 1. Compile expression with Groovy (v1 path) - 2. Load transpiled MalExpression (v2 path) - 3. Construct realistic sample data (non-trivial to prevent vacuous agreement) - 4. Run both paths with identical input - 5. Assert identical output: - - ExpressionParsingContext (scope, function, datatype, samples, downsampling) - - SampleFamily result (values, labels, entity descriptions) -``` - -### 9.2 Staleness Detection - -Properties files record SHA-256 hashes of upstream classes that have same-FQCN replacements. If upstream changes a class, the staleness test fails, forcing review of the replacement. - -### 9.3 Automatic Coverage - -New MAL/LAL YAML rules added to `server-starter/src/main/resources/` are automatically covered by the transpiler and comparison tests -- if the transpiler produces different results from Groovy, the build fails. - ---- - -## 10. Statistics - -| Metric | Count | -|--------|-------| -| MAL YAML files processed | 71 | -| MAL metric expressions transpiled | 1,254 | -| MAL filter expressions transpiled | 29 | -| LAL YAML files processed | 8 | -| LAL rules transpiled | 10 (6 unique after SHA-256 dedup) | -| Hierarchy rules replaced | 4 | -| Total generated Java classes | ~1,289 | -| Comparison test assertions | 1,300+ (MAL: 1,281, LAL: 19, hierarchy: 4 rules x multiple service pairs) | -| Lines of transpiler code (MAL) | ~1,230 | -| Lines of transpiler code (LAL) | ~950 | -| Runtime JAR removed | groovy-5.0.3.jar (~7 MB) | - ---- - -## 11. Risk Assessment - -| Risk | Mitigation | -|------|-----------| -| Transpiler misses an AST pattern | 1,300 dual-path comparison tests catch any divergence | -| New MAL/LAL expression uses unsupported Groovy syntax | Transpiler throws clear error at build time; new pattern must be added | -| Upstream SampleFamily/Spec changes break replacement | Staleness tests detect SHA-256 changes | -| Performance regression | Eliminated dynamic dispatch should only improve performance; benchmark with `MetricConvert` pipeline | -| Custom user MAL/LAL scripts | Users who extend default rules with custom Groovy scripts must follow the same syntax subset supported by the transpiler | - ---- - -## 12. Migration Timeline - -1. **Phase 1**: Add interfaces and functional interface overloads to existing `meter-analyzer` and `log-analyzer` (non-breaking, additive changes) -2. **Phase 2-3**: Implement MAL and LAL transpilers in new `mal-lal-v2/` modules -3. **Phase 4**: Implement v2 runtime loaders (modified `DSL.java`, `Expression.java`, `FilterExpression.java`) -4. **Phase 5**: Hierarchy v1/v2 module split -- refactor `server-core` to remove Groovy, create `hierarchy-v1/` (Groovy, checker-only) and `hierarchy-v2/` (Java lambdas, runtime) -5. **Phase 6**: Build comparison test suites -- `mal-lal-v1-v2-checker/` AND `hierarchy-v1-v2-checker/` -6. **Phase 7**: Switch `server-starter` from v1 to v2 for all three subsystems (MAL, LAL, hierarchy), remove Groovy from runtime classpath -7. **Phase 8**: Replace v2 manifest-based class loading with real ANTLR4 + Javassist compilers following the OAL pattern, enabling runtime compilation when configs change. Rename modules (drop `-v2` suffix), move v1/transpiler/checker to `test/script-compiler/`. diff --git a/docs/en/concepts-and-designs/lal.md b/docs/en/concepts-and-designs/lal.md index f843871cf818..4dff550d3dee 100644 --- a/docs/en/concepts-and-designs/lal.md +++ b/docs/en/concepts-and-designs/lal.md @@ -31,7 +31,7 @@ are cases where you may want the filter chain to stop earlier when specified con the remaining filter chain from where it's declared, and all the remaining components won't be executed at all. `abort` function serves as a fast-fail mechanism in LAL. -```groovy +``` filter { if (log.service == "TestingService") { // Don't waste resources on TestingServices abort {} // all remaining components won't be executed at all @@ -67,7 +67,7 @@ We can add tags like following: ] ``` And we can use this method to get the value of the tag key `TEST_KEY`. -```groovy +``` filter { if (tag("TEST_KEY") == "TEST_VALUE") { ... @@ -95,7 +95,7 @@ See examples below. #### `json` -```groovy +``` filter { json { abortOnFailure true // this is optional because it's default behaviour @@ -105,7 +105,7 @@ filter { #### `yaml` -```groovy +``` filter { yaml { abortOnFailure true // this is optional because it's default behaviour @@ -123,7 +123,7 @@ For unstructured logs, there are some `text` parsers for use. all the captured groups can be used later in the extractors or sinks. `regexp` returns a `boolean` indicating whether the log matches the pattern or not. -```groovy +``` filter { text { abortOnFailure true // this is optional because it's default behaviour @@ -181,7 +181,7 @@ dropped) and is used to associate with traces / metrics. not dropped) and is used to associate with traces / metrics. The parameter of `timestamp` can be a millisecond: -```groovy +``` filter { // ... parser @@ -191,7 +191,7 @@ filter { } ``` or a datetime string with a specified pattern: -```groovy +``` filter { // ... parser @@ -210,9 +210,7 @@ not dropped) and is used to associate with service. `tag` extracts the tags from the `parsed` result, and set them into the `LogData`. The form of this extractor should look something like this: `tag key1: value, key2: value2`. You may use the properties of `parsed` as both keys and values. -```groovy -import javax.swing.text.LayeredHighlighter - +``` filter { // ... parser @@ -242,7 +240,7 @@ log-analyzer: Examples are as follows: -```groovy +``` filter { // ... extractor { @@ -338,7 +336,7 @@ dropped) and is used to associate with TopNDatabaseStatement. An example of LAL to distinguish slow logs: -```groovy +``` filter { json{ } @@ -386,7 +384,7 @@ An example of JSON sent to OAP is as following: ``` Examples are as follows: -```groovy +``` filter { json { } @@ -447,7 +445,7 @@ final sampling result. See examples in [Enforcer](#enforcer). Examples 1, `rateLimit`: -```groovy +``` filter { // ... parser @@ -469,7 +467,7 @@ filter { Examples 2, `possibility`: -```groovy +``` filter { // ... parser @@ -492,7 +490,7 @@ filter { Dropper is a special sink, meaning that all logs are dropped without any exception. This is useful when you want to drop debugging logs. -```groovy +``` filter { // ... parser @@ -510,7 +508,7 @@ filter { Or if you have multiple filters, some of which are for extracting metrics, only one of them has to be persisted. -```groovy +``` filter { // filter A: this is for persistence // ... parser @@ -539,7 +537,7 @@ Enforcer is another special sink that forcibly samples the log. A typical use ca configured a sampler and want to save some logs forcibly, such as to save error logs even if the sampling mechanism has been configured. -```groovy +``` filter { // ... parser diff --git a/docs/en/concepts-and-designs/service-hierarchy-configuration.md b/docs/en/concepts-and-designs/service-hierarchy-configuration.md index 6aa0d40d56b2..7aac0e99fa9e 100644 --- a/docs/en/concepts-and-designs/service-hierarchy-configuration.md +++ b/docs/en/concepts-and-designs/service-hierarchy-configuration.md @@ -69,7 +69,7 @@ layer-levels: ### Auto Matching Rules - The auto matching rules are defined in the `auto-matching-rules` section. -- Use Groovy script to define the matching rules, the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean, +- The matching rules are expressions where the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean, which are used to match the relation between the upper service(u) and the lower service(l) on the different layers. - The default matching rules required the service name configured as SkyWalking default and follow the [Showcase](https://github.com/apache/skywalking-showcase). If you customized the service name in any layer, you should customize the related matching rules according your service name rules. diff --git a/docs/en/concepts-and-designs/service-hierarchy.md b/docs/en/concepts-and-designs/service-hierarchy.md index 8cd18bccda73..5f3c5144fcf1 100644 --- a/docs/en/concepts-and-designs/service-hierarchy.md +++ b/docs/en/concepts-and-designs/service-hierarchy.md @@ -45,7 +45,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### GENERAL On K8S_SERVICE - Rule name: `lower-short-name-remove-ns` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')) }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')) }` - Description: GENERAL.service.shortName == K8S_SERVICE.service.shortName without namespace - Matched Example: - GENERAL.service.name: `agent::songs` @@ -53,7 +53,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### GENERAL On APISIX - Rule name: `lower-short-name-remove-ns` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')) }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')) }` - Description: GENERAL.service.shortName == APISIX.service.shortName without namespace - Matched Example: - GENERAL.service.name: `agent::frontend` @@ -62,7 +62,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### VIRTUAL_DATABASE On MYSQL - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_DATABASE.service.shortName remove port == MYSQL.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_DATABASE.service.name: `mysql.skywalking-showcase.svc.cluster.local:3306` @@ -70,7 +70,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### VIRTUAL_DATABASE On POSTGRESQL - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_DATABASE.service.shortName remove port == POSTGRESQL.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_DATABASE.service.name: `psql.skywalking-showcase.svc.cluster.local:5432` @@ -78,7 +78,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### VIRTUAL_DATABASE On CLICKHOUSE - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_DATABASE.service.shortName remove port == CLICKHOUSE.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_DATABASE.service.name: `clickhouse.skywalking-showcase.svc.cluster.local:8123` @@ -87,7 +87,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### VIRTUAL_MQ On ROCKETMQ - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_MQ.service.shortName remove port == ROCKETMQ.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_MQ.service.name: `rocketmq.skywalking-showcase.svc.cluster.local:9876` @@ -95,7 +95,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### VIRTUAL_MQ On RABBITMQ - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_MQ.service.shortName remove port == RABBITMQ.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_MQ.service.name: `rabbitmq.skywalking-showcase.svc.cluster.local:5672` @@ -103,7 +103,7 @@ If you want to customize it according to your own needs, please refer to [Servic - #### VIRTUAL_MQ On KAFKA - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_MQ.service.shortName remove port == KAFKA.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_MQ.service.name: `kafka.skywalking-showcase.svc.cluster.local:9092` @@ -111,7 +111,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### VIRTUAL_MQ On PULSAR - Rule name: `lower-short-name-with-fqdn` -- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Matching expression: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` - Description: VIRTUAL_MQ.service.shortName remove port == PULSAR.service.shortName with fqdn suffix - Matched Example: - VIRTUAL_MQ.service.name: `pulsar.skywalking-showcase.svc.cluster.local:6650` @@ -119,7 +119,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### MESH On MESH_DP - Rule name: `name` -- Groovy script: `{ (u, l) -> u.name == l.name }` +- Matching expression: `{ (u, l) -> u.name == l.name }` - Description: MESH.service.name == MESH_DP.service.name - Matched Example: - MESH.service.name: `mesh-svr::songs.sample-services` @@ -127,7 +127,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### MESH On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: MESH.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - MESH.service.name: `mesh-svr::songs.sample-services` @@ -135,7 +135,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### MESH_DP On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: MESH_DP.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - MESH_DP.service.name: `mesh-svr::songs.sample-services` @@ -143,7 +143,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### MYSQL On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: MYSQL.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - MYSQL.service.name: `mysql::mysql.skywalking-showcase` @@ -151,7 +151,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### POSTGRESQL On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: POSTGRESQL.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - POSTGRESQL.service.name: `postgresql::psql.skywalking-showcase` @@ -159,7 +159,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### CLICKHOUSE On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: CLICKHOUSE.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - CLICKHOUSE.service.name: `clickhouse::clickhouse.skywalking-showcase` @@ -167,7 +167,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### NGINX On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: NGINX.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - NGINX.service.name: `nginx::nginx.skywalking-showcase` @@ -175,7 +175,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### APISIX On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: APISIX.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - APISIX.service.name: `APISIX::frontend.sample-services` @@ -183,7 +183,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### ROCKETMQ On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: ROCKETMQ.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - ROCKETMQ.service.name: `rocketmq::rocketmq.skywalking-showcase` @@ -191,7 +191,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### RABBITMQ On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: RABBITMQ.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - RABBITMQ.service.name: `rabbitmq::rabbitmq.skywalking-showcase` @@ -199,7 +199,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### KAFKA On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: KAFKA.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - KAFKA.service.name: `kafka::kafka.skywalking-showcase` @@ -207,7 +207,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### PULSAR On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: PULSAR.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - PULSAR.service.name: `pulsar::pulsar.skywalking-showcase` @@ -215,7 +215,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### SO11Y_OAP On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: SO11Y_OAP.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - SO11Y_OAP.service.name: `demo-oap.skywalking-showcase` @@ -223,7 +223,7 @@ If you want to customize it according to your own needs, please refer to [Servic #### KONG On K8S_SERVICE - Rule name: `short-name` -- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Matching expression: `{ (u, l) -> u.shortName == l.shortName }` - Description: KONG.service.shortName == K8S_SERVICE.service.shortName - Matched Example: - KONG.service.name: `kong::kong.skywalking-showcase` diff --git a/docs/menu.yml b/docs/menu.yml index adb2ee893649..72d08d1e3201 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -392,6 +392,8 @@ catalog: path: "/en/concepts-and-designs/ebpf-cpu-profiling" - name: "Diagnose Service Mesh Network Performance with eBPF" path: "/en/academy/diagnose-service-mesh-network-performance-with-ebpf" + - name: "DSL Compiler Design" + path: "/en/academy/dsl-compiler-design" - name: "FAQs" path: "/en/faq/readme" - name: "Contributing Guides" diff --git a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/meter/process/MeterProcessor.java b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/meter/process/MeterProcessor.java index d94afe0cdf4f..ea0900863a0c 100644 --- a/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/meter/process/MeterProcessor.java +++ b/oap-server/analyzer/agent-analyzer/src/main/java/org/apache/skywalking/oap/server/analyzer/provider/meter/process/MeterProcessor.java @@ -53,7 +53,7 @@ public class MeterProcessor { private final MeterProcessService processService; /** - * All of meters has been read. Using it to process groovy script. + * All of meters has been read. Using it to process MAL expressions. */ private final Map> meters = new HashMap<>(); From badac647a7ce27ef38066f253f9ef8d3322c5697 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 1 Mar 2026 10:33:55 +0800 Subject: [PATCH 11/64] Add run-e2e skill for local E2E test execution Provides a /run-e2e slash command with prerequisites (e2e CLI, swctl, yq install instructions), rebuild detection, test execution, and failure debugging workflow. Co-Authored-By: Claude Opus 4.6 --- .claude/skills/run-e2e/SKILL.md | 146 ++++++++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 .claude/skills/run-e2e/SKILL.md diff --git a/.claude/skills/run-e2e/SKILL.md b/.claude/skills/run-e2e/SKILL.md new file mode 100644 index 000000000000..5a49901a9ba0 --- /dev/null +++ b/.claude/skills/run-e2e/SKILL.md @@ -0,0 +1,146 @@ +--- +name: run-e2e +description: Run SkyWalking E2E tests locally +disable-model-invocation: true +argument-hint: "[test-case-path]" +--- + +# Run SkyWalking E2E Test + +Run an E2E test case using `skywalking-infra-e2e`. The user provides a test case path (e.g., `simple/jdk`, `storage/banyandb`, `alarm`). + +## Prerequisites + +All tools require **Go** installed. Check `.github/workflows/` for the exact `e2e` commit used in CI. + +### e2e CLI + +Built from [apache/skywalking-infra-e2e](https://github.com/apache/skywalking-infra-e2e), pinned by commit in CI: + +```bash +# Install the pinned commit +go install github.com/apache/skywalking-infra-e2e/cmd/e2e@ + +# Or clone and build locally (useful when debugging the e2e tool itself) +git clone https://github.com/apache/skywalking-infra-e2e.git +cd skywalking-infra-e2e +git checkout +make build +# binary is in bin/e2e — add to PATH or copy to $GOPATH/bin +``` + +### swctl, yq, and other tools + +E2E test cases run pre-install steps (see `setup.steps` in each `e2e.yaml`) that install tools into `/tmp/skywalking-infra-e2e/bin`. When running locally, you need these tools on your PATH. + +**swctl** — SkyWalking CLI, used in verify cases to query OAP's GraphQL API. Pinned at `SW_CTL_COMMIT` in `test/e2e-v2/script/env`: + +```bash +# Option 1: Use the install script (same as CI) +bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl +export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + +# Option 2: Build from source +go install github.com/apache/skywalking-cli/cmd/swctl@ +``` + +**yq** — YAML processor, used in verify cases: + +```bash +# Option 1: Use the install script +bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq +export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + +# Option 2: brew install yq (macOS) +``` + +**Other tools** (only needed for specific test cases): + +| Tool | Install script | Used by | +|------|---------------|---------| +| `kubectl` | `install.sh kubectl` | Kubernetes-based tests | +| `helm` | `install.sh helm` | Helm chart tests | +| `istioctl` | `install.sh istioctl` | Istio/service mesh tests | +| `etcdctl` | `install.sh etcdctl` | etcd cluster tests | + +All install scripts are at `test/e2e-v2/script/prepare/setup-e2e-shell/`. + +## Steps + +### 1. Determine the test case + +Resolve the user's argument to a full path under `test/e2e-v2/cases/`. If ambiguous, list matching directories and ask. + +```bash +ls test/e2e-v2/cases//e2e.yaml +``` + +### 2. Check if rebuild is needed + +Compare source file timestamps against the last build: + +```bash +# OAP server changes since last build +find oap-server apm-protocol -type f \( \ + -name "*.java" -o -name "*.yaml" -o -name "*.yml" -o \ + -name "*.json" -o -name "*.xml" -o -name "*.properties" -o \ + -name "*.proto" \ +\) -newer dist/apache-skywalking-apm-bin.tar.gz 2>/dev/null | head -5 + +# Test service changes since last build +find test/e2e-v2/java-test-service -type f \( \ + -name "*.java" -o -name "*.xml" -o -name "*.yaml" -o -name "*.yml" \ +\) -newer test/e2e-v2/java-test-service/e2e-service-provider/target/*.jar 2>/dev/null | head -5 +``` + +If files are found, warn the user and suggest rebuilding before running. + +### 3. Rebuild if needed (only with user confirmation) + +```bash +# Rebuild OAP +./mvnw clean package -Pall -Dmaven.test.skip && make docker + +# Rebuild test services +./mvnw -f test/e2e-v2/java-test-service/pom.xml clean package +``` + +### 4. Run the E2E test + +Set required environment variables and run: + +```bash +export SW_AGENT_JDK_VERSION=8 +e2e run -c test/e2e-v2/cases//e2e.yaml +``` + +### 5. If the test fails + +Do NOT run cleanup immediately. Instead: + +1. Check container logs: + ```bash + docker compose -f test/e2e-v2/cases//docker-compose.yml logs oap + docker compose -f test/e2e-v2/cases//docker-compose.yml logs provider + ``` + +2. Run verify separately (can retry after investigation): + ```bash + e2e verify -c test/e2e-v2/cases//e2e.yaml + ``` + +3. Only cleanup when done debugging: + ```bash + e2e cleanup -c test/e2e-v2/cases//e2e.yaml + ``` + +## Common test cases + +| Shorthand | Path | +|-----------|------| +| `simple/jdk` | `test/e2e-v2/cases/simple/jdk/` | +| `storage/banyandb` | `test/e2e-v2/cases/storage/banyandb/` | +| `storage/elasticsearch` | `test/e2e-v2/cases/storage/elasticsearch/` | +| `alarm` | `test/e2e-v2/cases/alarm/` | +| `log` | `test/e2e-v2/cases/log/` | +| `profiling/trace` | `test/e2e-v2/cases/profiling/trace/` | From e57381850a5d9610515f286b54421d4356c7fc53 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 1 Mar 2026 17:31:06 +0800 Subject: [PATCH 12/64] Fix LAL compiler gaps: tag(), safe nav, nested blocks, else-if chains, interpolated sampler IDs Address five critical gaps in the LAL v2 compiler that broke shipped production rules: 1. tag("LOG_KIND") in conditions now emits tagValue() helper instead of null 2. Safe navigation (?.) for method calls emits safeCall() helper to prevent NPE 3. Metrics, slowSql, sampledTrace, sampler/rateLimit blocks generate proper sub-consumer classes with BindingAware wiring 4. else-if chains build nested IfBlock AST nodes instead of dropping intermediate branches 5. GString interpolation in rateLimit IDs (e.g. "${log.service}:${parsed.code}") parsed into InterpolationPart segments and emitted as string concatenation Also fixes ProcessRegistry static calls to pass arguments through, and adds comprehensive tests (55 total: 35 generator + 20 parser) covering all gaps including production-like envoy-als, nginx, and k8s-service rule patterns. Co-Authored-By: Claude Opus 4.6 --- .../analyzer/compiler/LALClassGenerator.java | 636 +++++++++++++++++- .../log/analyzer/compiler/LALScriptModel.java | 77 ++- .../analyzer/compiler/LALScriptParser.java | 193 +++++- .../compiler/LALClassGeneratorTest.java | 458 +++++++++++++ .../compiler/LALScriptParserTest.java | 337 ++++++++++ 5 files changed, 1653 insertions(+), 48 deletions(-) diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java index a5cdcab939ea..be5e78d9efc7 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGenerator.java @@ -52,6 +52,22 @@ public final class LALClassGenerator { "org.apache.skywalking.oap.log.analyzer.dsl.Binding"; private static final String BINDING_PARSED = "org.apache.skywalking.oap.log.analyzer.dsl.Binding.Parsed"; + private static final String EXTRACTOR_SPEC = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.ExtractorSpec"; + private static final String SLOW_SQL_SPEC = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.slowsql.SlowSqlSpec"; + private static final String SAMPLED_TRACE_SPEC = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.extractor.sampledtrace.SampledTraceSpec"; + private static final String SINK_SPEC = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.SinkSpec"; + private static final String SAMPLER_SPEC = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.SamplerSpec"; + private static final String RATE_LIMITING_SAMPLER = + "org.apache.skywalking.oap.log.analyzer.dsl.spec.sink.sampler.RateLimitingSampler"; + private static final String SAMPLE_BUILDER = + EXTRACTOR_SPEC + "$SampleBuilder"; + private static final String PROCESS_REGISTRY = + "org.apache.skywalking.oap.meter.analyzer.dsl.registry.ProcessRegistry"; private final ClassPool classPool; @@ -82,7 +98,7 @@ public LalExpression compileFromModel(final LALScriptModel model) throws Excepti final List consumers = new ArrayList<>(); collectConsumers(model.getStatements(), consumers); - // Phase 2: Compile consumer classes + // Phase 2: Compile consumer classes (recursively handles sub-consumers) final List consumerInstances = new ArrayList<>(); for (int i = 0; i < consumers.size(); i++) { final String consumerName = className + "_C" + i; @@ -131,7 +147,7 @@ public LalExpression compileFromModel(final LALScriptModel model) throws Excepti // ==================== Consumer info ==================== - private static final class ConsumerInfo { + private static class ConsumerInfo { final String body; final String castType; final List subConsumers; @@ -141,6 +157,13 @@ private static final class ConsumerInfo { this.castType = castType; this.subConsumers = new ArrayList<>(); } + + ConsumerInfo(final String body, final String castType, + final List subConsumers) { + this.body = body; + this.castType = castType; + this.subConsumers = new ArrayList<>(subConsumers); + } } // ==================== Phase 1: Collect consumers ==================== @@ -184,19 +207,29 @@ private void collectConsumerFromStatement( } else if (stmt instanceof LALScriptModel.ExtractorBlock) { final LALScriptModel.ExtractorBlock block = (LALScriptModel.ExtractorBlock) stmt; + final ConsumerInfo info = new ConsumerInfo("", EXTRACTOR_SPEC); final StringBuilder sb = new StringBuilder(); - generateExtractorStatementsFlat(sb, block.getStatements()); - consumers.add(new ConsumerInfo(sb.toString(), - "org.apache.skywalking.oap.log.analyzer.dsl" - + ".spec.extractor.ExtractorSpec")); + final int[] subCounter = {0}; + final List extractorStmts = new ArrayList<>(); + for (final LALScriptModel.ExtractorStatement es : block.getStatements()) { + extractorStmts.add((LALScriptModel.FilterStatement) es); + } + generateExtractorBody(sb, extractorStmts, info, subCounter); + consumers.add(new ConsumerInfo(sb.toString(), EXTRACTOR_SPEC, + info.subConsumers)); } else if (stmt instanceof LALScriptModel.SinkBlock) { final LALScriptModel.SinkBlock sink = (LALScriptModel.SinkBlock) stmt; if (!sink.getStatements().isEmpty()) { + final ConsumerInfo info = new ConsumerInfo("", SINK_SPEC); final StringBuilder sb = new StringBuilder(); - generateSinkStatementsFlat(sb, sink.getStatements()); - consumers.add(new ConsumerInfo(sb.toString(), - "org.apache.skywalking.oap.log.analyzer.dsl" - + ".spec.sink.SinkSpec")); + final int[] subCounter = {0}; + final List sinkStmts = new ArrayList<>(); + for (final LALScriptModel.SinkStatement ss : sink.getStatements()) { + sinkStmts.add((LALScriptModel.FilterStatement) ss); + } + generateSinkBody(sb, sinkStmts, info, subCounter); + consumers.add(new ConsumerInfo(sb.toString(), SINK_SPEC, + info.subConsumers)); } } else if (stmt instanceof LALScriptModel.IfBlock) { final LALScriptModel.IfBlock ifBlock = (LALScriptModel.IfBlock) stmt; @@ -207,12 +240,14 @@ private void collectConsumerFromStatement( } } - // ==================== Flat code for consumer bodies ==================== + // ==================== Extractor body generation ==================== - private void generateExtractorStatementsFlat( + private void generateExtractorBody( final StringBuilder sb, - final List stmts) { - for (final LALScriptModel.ExtractorStatement stmt : stmts) { + final List stmts, + final ConsumerInfo parentInfo, + final int[] subCounter) { + for (final LALScriptModel.FilterStatement stmt : stmts) { if (stmt instanceof LALScriptModel.FieldAssignment) { final LALScriptModel.FieldAssignment field = (LALScriptModel.FieldAssignment) stmt; @@ -227,13 +262,423 @@ private void generateExtractorStatementsFlat( } sb.append(");\n"); } else if (stmt instanceof LALScriptModel.TagAssignment) { - final LALScriptModel.TagAssignment tag = - (LALScriptModel.TagAssignment) stmt; - generateTagAssignment(sb, tag); + generateTagAssignment(sb, (LALScriptModel.TagAssignment) stmt); + } else if (stmt instanceof LALScriptModel.IfBlock) { + generateIfBlockInBody(sb, (LALScriptModel.IfBlock) stmt, + parentInfo, subCounter, true); + } else if (stmt instanceof LALScriptModel.MetricsBlock) { + generateMetricsSubConsumer(sb, (LALScriptModel.MetricsBlock) stmt, + parentInfo, subCounter); + } else if (stmt instanceof LALScriptModel.SlowSqlBlock) { + generateSlowSqlSubConsumer(sb, (LALScriptModel.SlowSqlBlock) stmt, + parentInfo, subCounter); + } else if (stmt instanceof LALScriptModel.SampledTraceBlock) { + generateSampledTraceSubConsumer(sb, + (LALScriptModel.SampledTraceBlock) stmt, + parentInfo, subCounter); + } + } + } + + private void generateIfBlockInBody( + final StringBuilder sb, + final LALScriptModel.IfBlock ifBlock, + final ConsumerInfo parentInfo, + final int[] subCounter, + final boolean isExtractorContext) { + sb.append(" if ("); + generateCondition(sb, ifBlock.getCondition()); + sb.append(") {\n"); + if (isExtractorContext) { + generateExtractorBody(sb, ifBlock.getThenBranch(), parentInfo, subCounter); + } else { + generateSinkBody(sb, ifBlock.getThenBranch(), parentInfo, subCounter); + } + sb.append(" }\n"); + if (!ifBlock.getElseBranch().isEmpty()) { + sb.append(" else {\n"); + if (isExtractorContext) { + generateExtractorBody(sb, ifBlock.getElseBranch(), parentInfo, subCounter); + } else { + generateSinkBody(sb, ifBlock.getElseBranch(), parentInfo, subCounter); + } + sb.append(" }\n"); + } + } + + // ==================== Metrics sub-consumer ==================== + + private void generateMetricsSubConsumer( + final StringBuilder sb, + final LALScriptModel.MetricsBlock block, + final ConsumerInfo parentInfo, + final int[] subCounter) { + final int idx = subCounter[0]++; + final StringBuilder body = new StringBuilder(); + if (block.getName() != null) { + body.append(" _t.name(\"") + .append(escapeJava(block.getName())).append("\");\n"); + } + if (block.getTimestampValue() != null) { + body.append(" _t.timestamp("); + generateCastedValueAccess(body, block.getTimestampValue(), + block.getTimestampCast()); + body.append(");\n"); + } + if (!block.getLabels().isEmpty()) { + body.append(" { java.util.Map _labels = new java.util.LinkedHashMap();\n"); + for (final Map.Entry entry + : block.getLabels().entrySet()) { + body.append(" _labels.put(\"") + .append(escapeJava(entry.getKey())).append("\", "); + generateCastedValueAccess(body, entry.getValue().getValue(), + entry.getValue().getCastType()); + body.append(");\n"); + } + body.append(" _t.labels(_labels); }\n"); + } + if (block.getValue() != null) { + body.append(" _t.value("); + if ("Long".equals(block.getValueCast())) { + body.append("(double) toLong("); + generateValueAccess(body, block.getValue()); + body.append(")"); + } else if ("Integer".equals(block.getValueCast())) { + body.append("(double) toInt("); + generateValueAccess(body, block.getValue()); + body.append(")"); + } else { + // Number literal or untyped value — cast to double for Sample.value(double) + if (block.getValue().isNumberLiteral()) { + body.append("(double) ").append(block.getValue().getSegments().get(0)); + } else { + body.append("((Number) "); + generateValueAccess(body, block.getValue()); + body.append(").doubleValue()"); + } + } + body.append(");\n"); + } + + final ConsumerInfo sub = new ConsumerInfo(body.toString(), SAMPLE_BUILDER); + parentInfo.subConsumers.add(sub); + sb.append(" ((").append(PACKAGE_PREFIX) + .append("BindingAware) this._sub").append(idx) + .append(").setBinding(this.binding);\n"); + sb.append(" _t.metrics(this._sub").append(idx).append(");\n"); + } + + // ==================== SlowSql sub-consumer ==================== + + private void generateSlowSqlSubConsumer( + final StringBuilder sb, + final LALScriptModel.SlowSqlBlock block, + final ConsumerInfo parentInfo, + final int[] subCounter) { + final int idx = subCounter[0]++; + final StringBuilder body = new StringBuilder(); + if (block.getId() != null) { + body.append(" _t.id("); + generateCastedValueAccess(body, block.getId(), block.getIdCast()); + body.append(");\n"); + } + if (block.getStatement() != null) { + body.append(" _t.statement("); + generateCastedValueAccess(body, block.getStatement(), + block.getStatementCast()); + body.append(");\n"); + } + if (block.getLatency() != null) { + body.append(" _t.latency(Long.valueOf(toLong("); + generateValueAccess(body, block.getLatency()); + body.append(")));\n"); + } + + final ConsumerInfo sub = new ConsumerInfo(body.toString(), SLOW_SQL_SPEC); + parentInfo.subConsumers.add(sub); + sb.append(" ((").append(PACKAGE_PREFIX) + .append("BindingAware) this._sub").append(idx) + .append(").setBinding(this.binding);\n"); + sb.append(" _t.slowSql(this._sub").append(idx).append(");\n"); + } + + // ==================== SampledTrace sub-consumer ==================== + + private void generateSampledTraceSubConsumer( + final StringBuilder sb, + final LALScriptModel.SampledTraceBlock block, + final ConsumerInfo parentInfo, + final int[] subCounter) { + final int idx = subCounter[0]++; + final StringBuilder body = new StringBuilder(); + final ConsumerInfo sub = new ConsumerInfo("", SAMPLED_TRACE_SPEC); + final int[] innerSubCounter = {0}; + generateSampledTraceBody(body, block.getStatements(), sub, innerSubCounter); + + // Propagate any sub-sub-consumers + parentInfo.subConsumers.add(new ConsumerInfo(body.toString(), + SAMPLED_TRACE_SPEC, sub.subConsumers)); + sb.append(" ((").append(PACKAGE_PREFIX) + .append("BindingAware) this._sub").append(idx) + .append(").setBinding(this.binding);\n"); + sb.append(" _t.sampledTrace(this._sub").append(idx).append(");\n"); + } + + private void generateSampledTraceBody( + final StringBuilder sb, + final List stmts, + final ConsumerInfo parentInfo, + final int[] subCounter) { + for (final LALScriptModel.SampledTraceStatement stmt : stmts) { + if (stmt instanceof LALScriptModel.SampledTraceField) { + generateSampledTraceField(sb, (LALScriptModel.SampledTraceField) stmt); + } else if (stmt instanceof LALScriptModel.IfBlock) { + generateSampledTraceIfBlock(sb, (LALScriptModel.IfBlock) stmt, + parentInfo, subCounter); + } + } + } + + private void generateSampledTraceField( + final StringBuilder sb, + final LALScriptModel.SampledTraceField field) { + final String methodName; + switch (field.getFieldType()) { + case LATENCY: + methodName = "latency"; + sb.append(" _t.latency(Long.valueOf(toLong("); + generateValueAccess(sb, field.getValue()); + sb.append(")));\n"); + return; + case COMPONENT_ID: + methodName = "componentId"; + sb.append(" _t.componentId(toInt("); + generateValueAccess(sb, field.getValue()); + sb.append("));\n"); + return; + case URI: + methodName = "uri"; + break; + case REASON: + methodName = "reason"; + break; + case PROCESS_ID: + methodName = "processId"; + break; + case DEST_PROCESS_ID: + methodName = "destProcessId"; + break; + case DETECT_POINT: + methodName = "detectPoint"; + break; + case REPORT_SERVICE: + methodName = "reportService"; + break; + default: + return; + } + sb.append(" _t.").append(methodName).append("("); + generateCastedValueAccess(sb, field.getValue(), field.getCastType()); + sb.append(");\n"); + } + + private void generateSampledTraceIfBlock( + final StringBuilder sb, + final LALScriptModel.IfBlock ifBlock, + final ConsumerInfo parentInfo, + final int[] subCounter) { + sb.append(" if ("); + generateCondition(sb, ifBlock.getCondition()); + sb.append(") {\n"); + generateSampledTraceBodyFromFilterStmts(sb, ifBlock.getThenBranch(), + parentInfo, subCounter); + sb.append(" }\n"); + if (!ifBlock.getElseBranch().isEmpty()) { + sb.append(" else {\n"); + generateSampledTraceBodyFromFilterStmts(sb, ifBlock.getElseBranch(), + parentInfo, subCounter); + sb.append(" }\n"); + } + } + + private void generateSampledTraceBodyFromFilterStmts( + final StringBuilder sb, + final List stmts, + final ConsumerInfo parentInfo, + final int[] subCounter) { + for (final LALScriptModel.FilterStatement stmt : stmts) { + if (stmt instanceof LALScriptModel.FieldAssignment) { + // SampledTrace fields (processId, latency, etc.) are parsed as FieldAssignment + generateSampledTraceFieldFromAssignment(sb, + (LALScriptModel.FieldAssignment) stmt); + } else if (stmt instanceof LALScriptModel.IfBlock) { + generateSampledTraceIfBlock(sb, (LALScriptModel.IfBlock) stmt, + parentInfo, subCounter); + } + } + } + + private void generateSampledTraceFieldFromAssignment( + final StringBuilder sb, + final LALScriptModel.FieldAssignment fa) { + // Map FieldType to SampledTraceSpec methods + switch (fa.getFieldType()) { + case TIMESTAMP: + sb.append(" _t.latency(Long.valueOf(toLong("); + generateValueAccess(sb, fa.getValue()); + sb.append(")));\n"); + break; + default: + sb.append(" _t.").append(fa.getFieldType().name().toLowerCase()) + .append("("); + generateCastedValueAccess(sb, fa.getValue(), fa.getCastType()); + sb.append(");\n"); + break; + } + } + + // ==================== Sink body generation ==================== + + private void generateSinkBody( + final StringBuilder sb, + final List stmts, + final ConsumerInfo parentInfo, + final int[] subCounter) { + for (final LALScriptModel.FilterStatement stmt : stmts) { + if (stmt instanceof LALScriptModel.EnforcerStatement) { + sb.append(" _t.enforcer();\n"); + } else if (stmt instanceof LALScriptModel.DropperStatement) { + sb.append(" _t.dropper();\n"); + } else if (stmt instanceof LALScriptModel.SamplerBlock) { + generateSamplerSubConsumer(sb, (LALScriptModel.SamplerBlock) stmt, + parentInfo, subCounter); + } else if (stmt instanceof LALScriptModel.IfBlock) { + generateIfBlockInBody(sb, (LALScriptModel.IfBlock) stmt, + parentInfo, subCounter, false); + } + } + } + + // ==================== Sampler sub-consumer ==================== + + private void generateSamplerSubConsumer( + final StringBuilder sb, + final LALScriptModel.SamplerBlock block, + final ConsumerInfo parentInfo, + final int[] subCounter) { + final int idx = subCounter[0]++; + final StringBuilder body = new StringBuilder(); + final ConsumerInfo sub = new ConsumerInfo("", SAMPLER_SPEC); + final int[] innerSubCounter = {0}; + generateSamplerBody(body, block.getContents(), sub, innerSubCounter); + + parentInfo.subConsumers.add(new ConsumerInfo(body.toString(), + SAMPLER_SPEC, sub.subConsumers)); + sb.append(" ((").append(PACKAGE_PREFIX) + .append("BindingAware) this._sub").append(idx) + .append(").setBinding(this.binding);\n"); + sb.append(" _t.sampler(this._sub").append(idx).append(");\n"); + } + + private void generateSamplerBody( + final StringBuilder sb, + final List contents, + final ConsumerInfo parentInfo, + final int[] subCounter) { + for (final LALScriptModel.SamplerContent content : contents) { + if (content instanceof LALScriptModel.RateLimitBlock) { + generateRateLimitSubConsumer(sb, (LALScriptModel.RateLimitBlock) content, + parentInfo, subCounter); + } else if (content instanceof LALScriptModel.IfBlock) { + generateSamplerIfBlock(sb, (LALScriptModel.IfBlock) content, + parentInfo, subCounter); + } + } + } + + private void generateSamplerIfBlock( + final StringBuilder sb, + final LALScriptModel.IfBlock ifBlock, + final ConsumerInfo parentInfo, + final int[] subCounter) { + sb.append(" if ("); + generateCondition(sb, ifBlock.getCondition()); + sb.append(") {\n"); + generateSamplerBodyFromFilterStmts(sb, ifBlock.getThenBranch(), + parentInfo, subCounter); + sb.append(" }\n"); + if (!ifBlock.getElseBranch().isEmpty()) { + sb.append(" else {\n"); + generateSamplerBodyFromFilterStmts(sb, ifBlock.getElseBranch(), + parentInfo, subCounter); + sb.append(" }\n"); + } + } + + private void generateSamplerBodyFromFilterStmts( + final StringBuilder sb, + final List stmts, + final ConsumerInfo parentInfo, + final int[] subCounter) { + for (final LALScriptModel.FilterStatement stmt : stmts) { + if (stmt instanceof LALScriptModel.SamplerBlock) { + // SamplerBlock appears in if-branches inside a sampler + generateSamplerSubConsumerInline(sb, + (LALScriptModel.SamplerBlock) stmt, + parentInfo, subCounter); + } else if (stmt instanceof LALScriptModel.IfBlock) { + generateSamplerIfBlock(sb, (LALScriptModel.IfBlock) stmt, + parentInfo, subCounter); } } } + private void generateSamplerSubConsumerInline( + final StringBuilder sb, + final LALScriptModel.SamplerBlock block, + final ConsumerInfo parentInfo, + final int[] subCounter) { + // When a sampler block appears inside an if branch of a sampler, + // generate its contents inline + generateSamplerBody(sb, block.getContents(), parentInfo, subCounter); + } + + private void generateRateLimitSubConsumer( + final StringBuilder sb, + final LALScriptModel.RateLimitBlock block, + final ConsumerInfo parentInfo, + final int[] subCounter) { + final int idx = subCounter[0]++; + final String body = " _t.rpm(" + block.getRpm() + ");\n"; + final ConsumerInfo sub = new ConsumerInfo(body, RATE_LIMITING_SAMPLER); + parentInfo.subConsumers.add(sub); + sb.append(" ((").append(PACKAGE_PREFIX) + .append("BindingAware) this._sub").append(idx) + .append(").setBinding(this.binding);\n"); + + if (block.isIdInterpolated()) { + // Emit string concatenation for interpolated IDs + // e.g. "${log.service}:${parsed?.field}" → + // "" + String.valueOf(binding.log().getService()) + ":" + String.valueOf(...) + sb.append(" _t.rateLimit(\"\""); + for (final LALScriptModel.InterpolationPart part : block.getIdParts()) { + sb.append(" + "); + if (part.isLiteral()) { + sb.append("\"").append(escapeJava(part.getLiteral())).append("\""); + } else { + sb.append("String.valueOf("); + generateValueAccess(sb, part.getExpression()); + sb.append(")"); + } + } + sb.append(", this._sub").append(idx).append(");\n"); + } else { + sb.append(" _t.rateLimit(\"") + .append(escapeJava(block.getId())).append("\", this._sub") + .append(idx).append(");\n"); + } + } + private void generateTagAssignment(final StringBuilder sb, final LALScriptModel.TagAssignment tag) { final Map tags = tag.getTags(); @@ -262,18 +707,6 @@ private void generateTagAssignment(final StringBuilder sb, } } - private void generateSinkStatementsFlat( - final StringBuilder sb, - final List stmts) { - for (final LALScriptModel.SinkStatement stmt : stmts) { - if (stmt instanceof LALScriptModel.EnforcerStatement) { - sb.append(" _t.enforcer();\n"); - } else if (stmt instanceof LALScriptModel.DropperStatement) { - sb.append(" _t.dropper();\n"); - } - } - } - // ==================== Phase 2: Compile consumer classes ==================== private Object compileConsumerClass(final String className, @@ -294,17 +727,39 @@ private Object compileConsumerClass(final String className, "public " + BINDING + " getBinding() {" + " return this.binding; }", ctClass)); + // Add sub-consumer fields + for (int i = 0; i < info.subConsumers.size(); i++) { + ctClass.addField(CtField.make( + "public java.util.function.Consumer _sub" + i + ";", + ctClass)); + } + addHelperMethods(ctClass); final String method = "public void accept(Object arg) {\n" + " " + info.castType + " _t = (" + info.castType + ") arg;\n" + info.body + "}\n"; + + if (log.isDebugEnabled()) { + log.debug("LAL compile consumer {} body:\n{}", className, method); + } + ctClass.addMethod(CtNewMethod.make(method, ctClass)); final Class clazz = ctClass.toClass(LalExpressionPackageHolder.class); ctClass.detach(); - return clazz.getDeclaredConstructor().newInstance(); + final Object instance = clazz.getDeclaredConstructor().newInstance(); + + // Compile and wire sub-consumers + for (int i = 0; i < info.subConsumers.size(); i++) { + final String subName = className + "_S" + i; + final Object subInstance = compileConsumerClass( + subName, info.subConsumers.get(i)); + clazz.getField("_sub" + i).set(instance, subInstance); + } + + return instance; } // ==================== Phase 4: Generate execute method ==================== @@ -442,6 +897,32 @@ private void addHelperMethods(final CtClass ctClass) throws Exception { + " return ((Number) obj).doubleValue() != 0;" + " return true;" + "}", ctClass)); + + // tag() value lookup using Binding + ctClass.addMethod(CtNewMethod.make( + "private static String tagValue(" + + BINDING + " b, String key) {" + + " java.util.List dl = b.log().getTags().getDataList();" + + " for (int i = 0; i < dl.size(); i++) {" + + " org.apache.skywalking.apm.network.common.v3" + + ".KeyStringValuePair kv = " + + "(org.apache.skywalking.apm.network.common.v3" + + ".KeyStringValuePair) dl.get(i);" + + " if (key.equals(kv.getKey())) return kv.getValue();" + + " }" + + " return \"\";" + + "}", ctClass)); + + // Safe method call helper + ctClass.addMethod(CtNewMethod.make( + "private static Object safeCall(Object obj, String method) {" + + " if (obj == null) return null;" + + " if (\"toString\".equals(method)) return obj.toString();" + + " if (\"trim\".equals(method)) return obj.toString().trim();" + + " if (\"isEmpty\".equals(method))" + + " return Boolean.valueOf(obj.toString().isEmpty());" + + " return obj.toString();" + + "}", ctClass)); } // ==================== Conditions ==================== @@ -597,6 +1078,45 @@ private void generateValueAccessObj(final StringBuilder sb, private void generateValueAccess(final StringBuilder sb, final LALScriptModel.ValueAccess value) { + // Handle function call primaries (e.g., tag("LOG_KIND")) + if (value.getFunctionCallName() != null) { + if ("tag".equals(value.getFunctionCallName()) + && !value.getFunctionCallArgs().isEmpty()) { + // tag("KEY") → tagValue(binding, "KEY") + sb.append("tagValue(binding, \""); + final String key = value.getFunctionCallArgs().get(0) + .getValue().getSegments().get(0); + sb.append(escapeJava(key)).append("\")"); + } else { + // Unknown function call — emit null for safety + sb.append("null"); + } + return; + } + + // Handle string/number literals + if (value.isStringLiteral() && value.getChain().isEmpty()) { + sb.append("\"").append(escapeJava(value.getSegments().get(0))) + .append("\""); + return; + } + if (value.isNumberLiteral() && value.getChain().isEmpty()) { + final String num = value.getSegments().get(0); + // Box number literals so Javassist resolves Object-param methods + if (num.contains(".")) { + sb.append("Double.valueOf(").append(num).append(")"); + } else { + sb.append("Integer.valueOf(").append(num).append(")"); + } + return; + } + + // Handle ProcessRegistry static calls + if (value.isProcessRegistryRef()) { + generateProcessRegistryCall(sb, value); + return; + } + String current; if (value.isParsedRef()) { current = "binding.parsed()"; @@ -622,17 +1142,71 @@ private void generateValueAccess(final StringBuilder sb, if (seg instanceof LALScriptModel.FieldSegment) { final String name = ((LALScriptModel.FieldSegment) seg).getName(); + // getAt() already handles null → null, so safe nav is free current = "getAt(" + current + ", \"" + escapeJava(name) + "\")"; } else if (seg instanceof LALScriptModel.MethodSegment) { final LALScriptModel.MethodSegment ms = (LALScriptModel.MethodSegment) seg; - current = current + "." + ms.getName() + "()"; + if (ms.isSafeNav()) { + // Safe navigation: null-safe method call + current = "safeCall(" + current + ", \"" + + escapeJava(ms.getName()) + "\")"; + } else { + if (ms.getArguments().isEmpty()) { + current = current + "." + ms.getName() + "()"; + } else { + current = current + "." + ms.getName() + "(" + + generateMethodArgs(ms.getArguments()) + ")"; + } + } } } sb.append(current); } + private void generateProcessRegistryCall( + final StringBuilder sb, + final LALScriptModel.ValueAccess value) { + final List chain = value.getChain(); + if (chain.isEmpty()) { + sb.append("null"); + return; + } + // Expect exactly one method segment: ProcessRegistry.methodName(args) + final LALScriptModel.ValueAccessSegment seg = chain.get(0); + if (seg instanceof LALScriptModel.MethodSegment) { + final LALScriptModel.MethodSegment ms = + (LALScriptModel.MethodSegment) seg; + sb.append(PROCESS_REGISTRY).append(".") + .append(ms.getName()).append("("); + final List args = ms.getArguments(); + for (int i = 0; i < args.size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateCastedValueAccess(sb, + args.get(i).getValue(), args.get(i).getCastType()); + } + sb.append(")"); + } else { + sb.append("null"); + } + } + + private String generateMethodArgs( + final List args) { + final StringBuilder sb = new StringBuilder(); + for (int i = 0; i < args.size(); i++) { + if (i > 0) { + sb.append(", "); + } + generateCastedValueAccess(sb, + args.get(i).getValue(), args.get(i).getCastType()); + } + return sb.toString(); + } + // ==================== Utilities ==================== private static String escapeJava(final String s) { diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java index cc3d1d3df04e..8609deb66ded 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptModel.java @@ -245,12 +245,44 @@ public interface SamplerContent { @Getter public static final class RateLimitBlock implements SamplerContent { private final String id; + private final List idParts; private final long rpm; - public RateLimitBlock(final String id, final long rpm) { + public RateLimitBlock(final String id, + final List idParts, + final long rpm) { this.id = id; + this.idParts = idParts != null + ? Collections.unmodifiableList(idParts) : Collections.emptyList(); this.rpm = rpm; } + + public boolean isIdInterpolated() { + return !idParts.isEmpty(); + } + } + + @Getter + public static final class InterpolationPart { + private final String literal; + private final ValueAccess expression; + + private InterpolationPart(final String literal, final ValueAccess expression) { + this.literal = literal; + this.expression = expression; + } + + public static InterpolationPart ofLiteral(final String text) { + return new InterpolationPart(text, null); + } + + public static InterpolationPart ofExpression(final ValueAccess expr) { + return new InterpolationPart(null, expr); + } + + public boolean isLiteral() { + return literal != null; + } } public static final class EnforcerStatement implements SinkStatement, FilterStatement { @@ -341,17 +373,41 @@ public static final class ValueAccess { private final List segments; private final boolean parsedRef; private final boolean logRef; + private final boolean processRegistryRef; + private final boolean stringLiteral; + private final boolean numberLiteral; private final List chain; + private final String functionCallName; + private final List functionCallArgs; public ValueAccess(final List segments, final boolean parsedRef, final boolean logRef, final List chain) { + this(segments, parsedRef, logRef, false, false, false, + chain, null, Collections.emptyList()); + } + + public ValueAccess(final List segments, + final boolean parsedRef, + final boolean logRef, + final boolean processRegistryRef, + final boolean stringLiteral, + final boolean numberLiteral, + final List chain, + final String functionCallName, + final List functionCallArgs) { this.segments = Collections.unmodifiableList(segments); this.parsedRef = parsedRef; this.logRef = logRef; + this.processRegistryRef = processRegistryRef; + this.stringLiteral = stringLiteral; + this.numberLiteral = numberLiteral; this.chain = chain != null ? Collections.unmodifiableList(chain) : Collections.emptyList(); + this.functionCallName = functionCallName; + this.functionCallArgs = functionCallArgs != null + ? Collections.unmodifiableList(functionCallArgs) : Collections.emptyList(); } public String toPathString() { @@ -359,6 +415,17 @@ public String toPathString() { } } + @Getter + public static final class FunctionArg { + private final ValueAccess value; + private final String castType; + + public FunctionArg(final ValueAccess value, final String castType) { + this.value = value; + this.castType = castType; + } + } + public interface ValueAccessSegment { } @@ -376,12 +443,14 @@ public FieldSegment(final String name, final boolean safeNav) { @Getter public static final class MethodSegment implements ValueAccessSegment { private final String name; - private final List arguments; + private final List arguments; private final boolean safeNav; - public MethodSegment(final String name, final List arguments, final boolean safeNav) { + public MethodSegment(final String name, final List arguments, + final boolean safeNav) { this.name = name; - this.arguments = Collections.unmodifiableList(arguments); + this.arguments = arguments != null + ? Collections.unmodifiableList(arguments) : Collections.emptyList(); this.safeNav = safeNav; } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java index 9bd9d241c1b5..2793ed4dde84 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParser.java @@ -29,6 +29,7 @@ import org.apache.skywalking.lal.rt.grammar.LALLexer; import org.apache.skywalking.lal.rt.grammar.LALParser; import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.AbortStatement; +import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.InterpolationPart; import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.CompareOp; import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.ComparisonCondition; import org.apache.skywalking.oap.log.analyzer.compiler.LALScriptModel.Condition; @@ -428,7 +429,8 @@ private static SamplerBlock visitSamplerBlock(final LALParser.SamplerBlockContex for (final LALParser.RateLimitBlockContext rlc : ctx.samplerContent().rateLimitBlock()) { final String id = stripQuotes(rlc.rateLimitId().getText()); final long rpm = Long.parseLong(rlc.rateLimitContent().NUMBER().getText()); - contents.add(new RateLimitBlock(id, rpm)); + final List idParts = parseInterpolation(id); + contents.add(new RateLimitBlock(id, idParts, rpm)); } for (final LALParser.IfStatementContext isc : ctx.samplerContent().ifStatement()) { contents.add((SamplerContent) visitIfStatement(isc)); @@ -439,17 +441,45 @@ private static SamplerBlock visitSamplerBlock(final LALParser.SamplerBlockContex // ==================== If statement ==================== private static IfBlock visitIfStatement(final LALParser.IfStatementContext ctx) { - final Condition condition = visitCondition(ctx.condition(0)); - final List thenBranch = visitIfBody(ctx.ifBody(0)); - - List elseBranch = null; - // Handle else-if and else branches + final int condCount = ctx.condition().size(); final int bodyCount = ctx.ifBody().size(); - if (bodyCount > 1) { - elseBranch = visitIfBody(ctx.ifBody(bodyCount - 1)); + // Whether there is a trailing else (no condition) block + final boolean hasElse = bodyCount > condCount; + + // Build the chain from the last else-if backwards. + // For: if(A){b0} else if(B){b1} else if(C){b2} else{b3} + // condCount=3, bodyCount=4, hasElse=true + // Result: IfBlock(A, b0, IfBlock(B, b1, IfBlock(C, b2, b3))) + + // Start from the innermost else-if (last condition) + List trailingElse = hasElse + ? visitIfBody(ctx.ifBody(bodyCount - 1)) : null; + + // Build from the last condition backwards to index 1 + IfBlock nested = null; + for (int i = condCount - 1; i >= 1; i--) { + final Condition cond = visitCondition(ctx.condition(i)); + final List body = visitIfBody(ctx.ifBody(i)); + final List elsePart; + if (nested != null) { + elsePart = List.of(nested); + } else { + elsePart = trailingElse; + } + nested = new IfBlock(cond, body, elsePart); + } + + // Build the outermost if block (index 0) + final Condition topCond = visitCondition(ctx.condition(0)); + final List topBody = visitIfBody(ctx.ifBody(0)); + final List topElse; + if (nested != null) { + topElse = List.of(nested); + } else { + topElse = trailingElse; } - return new IfBlock(condition, thenBranch, elseBranch); + return new IfBlock(topCond, topBody, topElse); } private static List visitIfBody(final LALParser.IfBodyContext ctx) { @@ -541,8 +571,11 @@ private static Condition makeComparison( if (leftCtx instanceof LALParser.CondFunctionCallContext) { final LALParser.FunctionInvocationContext fi = ((LALParser.CondFunctionCallContext) leftCtx).functionInvocation(); + final String funcName = fi.functionName().getText(); + final List funcArgs = visitFunctionArgs(fi); final ValueAccess left = new ValueAccess( - List.of(fi.getText()), false, false, List.of()); + List.of(fi.getText()), false, false, false, false, false, + List.of(), funcName, funcArgs); return new ComparisonCondition(left, null, op, visitConditionExprAsValue(rightCtx)); } @@ -582,6 +615,16 @@ private static Condition visitConditionExprAsCondition( final String cast = va.typeCast() != null ? extractCastType(va.typeCast()) : null; return new ExprCondition(visitValueAccess(va.valueAccess()), cast); } + if (ctx instanceof LALParser.CondFunctionCallContext) { + final LALParser.FunctionInvocationContext fi = + ((LALParser.CondFunctionCallContext) ctx).functionInvocation(); + final String funcName = fi.functionName().getText(); + final List funcArgs = visitFunctionArgs(fi); + final ValueAccess va = new ValueAccess( + List.of(fi.getText()), false, false, false, false, false, + List.of(), funcName, funcArgs); + return new ExprCondition(va, null); + } return new ExprCondition( new ValueAccess(List.of(ctx.getText()), false, false, List.of()), null); } @@ -592,6 +635,11 @@ private static ValueAccess visitValueAccess(final LALParser.ValueAccessContext c final List segments = new ArrayList<>(); boolean parsedRef = false; boolean logRef = false; + boolean processRegistryRef = false; + boolean stringLiteral = false; + boolean numberLiteral = false; + String functionCallName = null; + List functionCallArgs = null; final LALParser.ValueAccessPrimaryContext primary = ctx.valueAccessPrimary(); if (primary instanceof LALParser.ValueParsedContext) { @@ -601,17 +649,22 @@ private static ValueAccess visitValueAccess(final LALParser.ValueAccessContext c logRef = true; segments.add("log"); } else if (primary instanceof LALParser.ValueProcessRegistryContext) { + processRegistryRef = true; segments.add("ProcessRegistry"); } else if (primary instanceof LALParser.ValueIdentifierContext) { segments.add(((LALParser.ValueIdentifierContext) primary).IDENTIFIER().getText()); } else if (primary instanceof LALParser.ValueStringContext) { + stringLiteral = true; segments.add(stripQuotes( ((LALParser.ValueStringContext) primary).STRING().getText())); } else if (primary instanceof LALParser.ValueNumberContext) { + numberLiteral = true; segments.add(((LALParser.ValueNumberContext) primary).NUMBER().getText()); } else if (primary instanceof LALParser.ValueFunctionCallContext) { final LALParser.FunctionInvocationContext fi = ((LALParser.ValueFunctionCallContext) primary).functionInvocation(); + functionCallName = fi.functionName().getText(); + functionCallArgs = visitFunctionArgs(fi); segments.add(fi.getText()); } else { segments.add(primary.getText()); @@ -634,17 +687,57 @@ private static ValueAccess visitValueAccess(final LALParser.ValueAccessContext c ((LALParser.SegmentMethodContext) seg).functionInvocation(); segments.add(fi.functionName().getText() + "()"); chain.add(new LALScriptModel.MethodSegment( - fi.functionName().getText(), List.of(), false)); + fi.functionName().getText(), visitFunctionArgs(fi), false)); } else if (seg instanceof LALParser.SegmentSafeMethodContext) { final LALParser.FunctionInvocationContext fi = ((LALParser.SegmentSafeMethodContext) seg).functionInvocation(); segments.add(fi.functionName().getText() + "()"); chain.add(new LALScriptModel.MethodSegment( - fi.functionName().getText(), List.of(), true)); + fi.functionName().getText(), visitFunctionArgs(fi), true)); } } - return new ValueAccess(segments, parsedRef, logRef, chain); + return new ValueAccess(segments, parsedRef, logRef, + processRegistryRef, stringLiteral, numberLiteral, + chain, functionCallName, functionCallArgs); + } + + private static List visitFunctionArgs( + final LALParser.FunctionInvocationContext fi) { + if (fi.functionArgList() == null) { + return List.of(); + } + final List args = new ArrayList<>(); + for (final LALParser.FunctionArgContext fac : fi.functionArgList().functionArg()) { + if (fac.valueAccess() != null) { + final ValueAccess va = visitValueAccess(fac.valueAccess()); + final String cast = fac.typeCast() != null + ? extractCastType(fac.typeCast()) : null; + args.add(new LALScriptModel.FunctionArg(va, cast)); + } else if (fac.STRING() != null) { + final String val = stripQuotes(fac.STRING().getText()); + final ValueAccess va = new ValueAccess( + List.of(val), false, false, true, true, false, + List.of(), null, null); + args.add(new LALScriptModel.FunctionArg(va, null)); + } else if (fac.NUMBER() != null) { + final ValueAccess va = new ValueAccess( + List.of(fac.NUMBER().getText()), false, false, + false, false, true, List.of(), null, null); + args.add(new LALScriptModel.FunctionArg(va, null)); + } else if (fac.boolValue() != null) { + final ValueAccess va = new ValueAccess( + List.of(fac.boolValue().getText()), false, false, + false, false, false, List.of(), null, null); + args.add(new LALScriptModel.FunctionArg(va, null)); + } else { + // NULL + final ValueAccess va = new ValueAccess( + List.of("null"), false, false, List.of()); + args.add(new LALScriptModel.FunctionArg(va, null)); + } + } + return args; } private static String resolveValueAsString(final LALParser.ValueAccessContext ctx) { @@ -694,4 +787,78 @@ private static String truncate(final String s, final int maxLen) { } return s.substring(0, maxLen) + "..."; } + + // ==================== GString interpolation ==================== + + /** + * Parses Groovy-style GString interpolation in a string. + * E.g. {@code "${log.service}:${parsed?.field}"} produces + * [expr(log.service), literal(":"), expr(parsed?.field)]. + * + * @return list of parts, or {@code null} if no interpolation found + */ + static List parseInterpolation(final String s) { + if (!s.contains("${")) { + return null; + } + final List parts = new ArrayList<>(); + int pos = 0; + while (pos < s.length()) { + final int start = s.indexOf("${", pos); + if (start < 0) { + // Remaining literal text + if (pos < s.length()) { + parts.add(InterpolationPart.ofLiteral(s.substring(pos))); + } + break; + } + // Literal text before ${ + if (start > pos) { + parts.add(InterpolationPart.ofLiteral(s.substring(pos, start))); + } + // Find matching closing brace, respecting nesting + int depth = 1; + int i = start + 2; + while (i < s.length() && depth > 0) { + final char c = s.charAt(i); + if (c == '{') { + depth++; + } else if (c == '}') { + depth--; + } + i++; + } + if (depth != 0) { + throw new IllegalArgumentException( + "Unclosed interpolation in: " + s); + } + final String expr = s.substring(start + 2, i - 1); + // Parse the expression as a valueAccess through ANTLR + parts.add(InterpolationPart.ofExpression(parseValueAccessExpr(expr))); + pos = i; + } + return parts; + } + + /** + * Parses a standalone valueAccess expression string by wrapping it in + * a minimal LAL script and extracting the parsed ValueAccess. + */ + private static ValueAccess parseValueAccessExpr(final String expr) { + // Wrap in: filter { if (EXPR) { sink {} } } + // The expression becomes a condition, parsed as ExprCondition + // whose ValueAccess is what we want. + final String wrapper = "filter { if (" + expr + ") { sink {} } }"; + final LALScriptModel model = parse(wrapper); + final IfBlock ifBlock = (IfBlock) model.getStatements().get(0); + final LALScriptModel.Condition cond = ifBlock.getCondition(); + if (cond instanceof ExprCondition) { + return ((ExprCondition) cond).getExpr(); + } + if (cond instanceof ComparisonCondition) { + return ((ComparisonCondition) cond).getLeft(); + } + throw new IllegalArgumentException( + "Cannot parse interpolation expression: " + expr); + } } diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java index 11cee26a833c..a1a10209e5b7 100644 --- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALClassGeneratorTest.java @@ -22,8 +22,11 @@ import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; class LALClassGeneratorTest { @@ -127,4 +130,459 @@ void invalidStatementInFilterThrows() { assertThrows(Exception.class, () -> generator.compile("filter { invalid {} }")); } + + // ==================== tag() function in conditions ==================== + + @Test + void compileTagFunctionInCondition() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " if (tag(\"LOG_KIND\") == \"SLOW_SQL\") {\n" + + " sink {}\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void generateSourceTagFunctionEmitsTagValue() { + final String source = generator.generateSource( + "filter {\n" + + " if (tag(\"LOG_KIND\") == \"SLOW_SQL\") {\n" + + " sink {}\n" + + " }\n" + + "}"); + // Should use tagValue helper, not emit null + assertTrue(source.contains("tagValue(binding, \"LOG_KIND\")"), + "Expected tagValue call but got: " + source); + assertTrue(source.contains("SLOW_SQL")); + } + + @Test + void compileTagFunctionNestedInExtractor() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " if (tag(\"LOG_KIND\") == \"NET_PROFILING_SAMPLED_TRACE\") {\n" + + " service parsed.service as String\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== Safe navigation ==================== + + @Test + void compileSafeNavigationFieldAccess() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed?.response?.service as String\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileSafeNavigationMethodCalls() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service parsed?.flags?.toString()?.trim() as String\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void generateSourceSafeNavMethodEmitsSafeCall() { + final String source = generator.generateSource( + "filter {\n" + + " if (parsed?.flags?.toString()) {\n" + + " sink {}\n" + + " }\n" + + "}"); + // Safe method calls should use safeCall helper + assertTrue(source.contains("safeCall("), + "Expected safeCall for safe nav method but got: " + source); + } + + // ==================== ProcessRegistry static calls ==================== + + @Test + void compileProcessRegistryCall() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service ProcessRegistry.generateVirtualLocalProcess(" + + "parsed.service as String, parsed.serviceInstance as String" + + ") as String\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileProcessRegistryWithThreeArgs() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service ProcessRegistry.generateVirtualRemoteProcess(" + + "parsed.service as String, parsed.serviceInstance as String, " + + "parsed.address as String) as String\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== Metrics block ==================== + + @Test + void compileMetricsBlock() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " metrics {\n" + + " timestamp log.timestamp as Long\n" + + " labels level: parsed.level, service: log.service\n" + + " name \"nginx_error_log_count\"\n" + + " value 1\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== SlowSql block ==================== + + @Test + void compileSlowSqlBlock() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " slowSql {\n" + + " id parsed.id as String\n" + + " statement parsed.statement as String\n" + + " latency parsed.query_time as Long\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== SampledTrace block ==================== + + @Test + void compileSampledTraceBlock() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " sampledTrace {\n" + + " latency parsed.latency as Long\n" + + " uri parsed.uri as String\n" + + " reason parsed.reason as String\n" + + " detectPoint parsed.detect_point as String\n" + + " componentId 49\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileSampledTraceWithIfBlocks() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " sampledTrace {\n" + + " latency parsed.latency as Long\n" + + " if (parsed.client_process.process_id as String != \"\") {\n" + + " processId parsed.client_process.process_id as String\n" + + " } else {\n" + + " processId parsed.fallback as String\n" + + " }\n" + + " detectPoint parsed.detect_point as String\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== Sampler / rateLimit ==================== + + @Test + void compileSamplerWithRateLimit() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit('service:error') {\n" + + " rpm 6000\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileSamplerWithInterpolatedId() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit(\"${log.service}:${parsed.code}\") {\n" + + " rpm 6000\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void parseInterpolatedIdParts() { + // Verify the parser correctly splits interpolated strings + final java.util.List parts = + LALScriptParser.parseInterpolation( + "${log.service}:${parsed.code}"); + assertNotNull(parts); + // expr, literal ":", expr + assertEquals(3, parts.size()); + assertFalse(parts.get(0).isLiteral()); + assertTrue(parts.get(0).getExpression().isLogRef()); + assertTrue(parts.get(1).isLiteral()); + assertEquals(":", parts.get(1).getLiteral()); + assertFalse(parts.get(2).isLiteral()); + assertTrue(parts.get(2).getExpression().isParsedRef()); + } + + @Test + void compileSamplerWithSafeNavInterpolatedId() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit(\"${log.service}:${parsed?.commonProperties?.responseFlags?.toString()}\") {\n" + + " rpm 6000\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileSamplerWithIfAndRateLimit() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " sink {\n" + + " sampler {\n" + + " if (parsed?.error) {\n" + + " rateLimit('svc:err') {\n" + + " rpm 6000\n" + + " }\n" + + " } else {\n" + + " rateLimit('svc:ok') {\n" + + " rpm 3000\n" + + " }\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + // ==================== If blocks in extractor/sink ==================== + + @Test + void compileIfInsideExtractor() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " if (parsed?.status) {\n" + + " tag 'http.status_code': parsed.status\n" + + " }\n" + + " tag 'response.flag': parsed.flags\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileIfInsideExtractorWithTagCondition() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " if (tag(\"LOG_KIND\") == \"NET_PROFILING\") {\n" + + " service parsed.service as String\n" + + " layer parsed.layer as String\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== Complex production-like rules ==================== + + @Test + void compileNginxAccessLogRule() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " if (tag(\"LOG_KIND\") == \"NGINX_ACCESS_LOG\") {\n" + + " text {\n" + + " regexp '.+\"(?.+)\"(?\\\\d{3}).+'\n" + + " }\n" + + " extractor {\n" + + " if (parsed.status) {\n" + + " tag 'http.status_code': parsed.status\n" + + " }\n" + + " }\n" + + " sink {}\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileSlowSqlProductionRule() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " if (tag(\"LOG_KIND\") == \"SLOW_SQL\") {\n" + + " layer parsed.layer as String\n" + + " service parsed.service as String\n" + + " timestamp parsed.time as String\n" + + " slowSql {\n" + + " id parsed.id as String\n" + + " statement parsed.statement as String\n" + + " latency parsed.query_time as Long\n" + + " }\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileEnvoyAlsAbortRule() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " if (parsed?.response?.responseCode?.value as Integer < 400" + + " && !parsed?.commonProperties?.responseFlags?.toString()?.trim()) {\n" + + " abort {}\n" + + " }\n" + + " extractor {\n" + + " if (parsed?.response?.responseCode) {\n" + + " tag 'status.code': parsed?.response?.responseCode?.value\n" + + " }\n" + + " tag 'response.flag': parsed?.commonProperties?.responseFlags\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + // ==================== Else-if chain ==================== + + @Test + void compileElseIfChain() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " if (parsed.a) {\n" + + " sink {}\n" + + " } else if (parsed.b) {\n" + + " sink {}\n" + + " } else if (parsed.c) {\n" + + " sink {}\n" + + " } else {\n" + + " sink {}\n" + + " }\n" + + "}"); + assertNotNull(expr); + } + + @Test + void compileElseIfInSampledTrace() throws Exception { + final LalExpression expr = generator.compile( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " sampledTrace {\n" + + " latency parsed.latency as Long\n" + + " if (parsed.client_process.process_id as String != \"\") {\n" + + " processId parsed.client_process.process_id as String\n" + + " } else if (parsed.client_process.local as Boolean) {\n" + + " processId ProcessRegistry.generateVirtualLocalProcess(" + + "parsed.service as String, parsed.serviceInstance as String) as String\n" + + " } else {\n" + + " processId ProcessRegistry.generateVirtualRemoteProcess(" + + "parsed.service as String, parsed.serviceInstance as String, " + + "parsed.client_process.address as String) as String\n" + + " }\n" + + " detectPoint parsed.detect_point as String\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + assertNotNull(expr); + } + + @Test + void generateSourceElseIfEmitsNestedBranches() { + final String source = generator.generateSource( + "filter {\n" + + " if (parsed.a) {\n" + + " sink {}\n" + + " } else if (parsed.b) {\n" + + " sink {}\n" + + " } else {\n" + + " sink {}\n" + + " }\n" + + "}"); + // The else-if should produce a nested if inside else + assertTrue(source.contains("else"), + "Expected else branch but got: " + source); + // Both condition branches should appear + int ifCount = 0; + for (int i = 0; i < source.length() - 2; i++) { + if (source.substring(i, i + 3).equals("if ")) { + ifCount++; + } + } + assertTrue(ifCount >= 2, + "Expected at least 2 if-conditions for else-if chain but got " + + ifCount + " in: " + source); + } } diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java index 5c29397f8c4c..8760ca4f6c0b 100644 --- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/compiler/LALScriptParserTest.java @@ -20,6 +20,7 @@ import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertInstanceOf; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertThrows; @@ -165,6 +166,65 @@ void parseSinkWithSampler() { assertEquals(6000, rateLimit.getRpm()); } + @Test + void parseInterpolatedRateLimitId() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit(\"${log.service}:${parsed.code}\") {\n" + + " rpm 3000\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + + final LALScriptModel.SinkBlock sink = + (LALScriptModel.SinkBlock) model.getStatements().get(0); + final LALScriptModel.SamplerBlock sampler = + (LALScriptModel.SamplerBlock) sink.getStatements().get(0); + final LALScriptModel.RateLimitBlock rl = + (LALScriptModel.RateLimitBlock) sampler.getContents().get(0); + + assertTrue(rl.isIdInterpolated()); + assertEquals(3, rl.getIdParts().size()); + + // Part 0: expression ${log.service} + assertFalse(rl.getIdParts().get(0).isLiteral()); + assertTrue(rl.getIdParts().get(0).getExpression().isLogRef()); + + // Part 1: literal ":" + assertTrue(rl.getIdParts().get(1).isLiteral()); + assertEquals(":", rl.getIdParts().get(1).getLiteral()); + + // Part 2: expression ${parsed.code} + assertFalse(rl.getIdParts().get(2).isLiteral()); + assertTrue(rl.getIdParts().get(2).getExpression().isParsedRef()); + } + + @Test + void parsePlainRateLimitIdNotInterpolated() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " sink {\n" + + " sampler {\n" + + " rateLimit('service:error') {\n" + + " rpm 6000\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + + final LALScriptModel.SinkBlock sink = + (LALScriptModel.SinkBlock) model.getStatements().get(0); + final LALScriptModel.SamplerBlock sampler = + (LALScriptModel.SamplerBlock) sink.getStatements().get(0); + final LALScriptModel.RateLimitBlock rl = + (LALScriptModel.RateLimitBlock) sampler.getContents().get(0); + + assertFalse(rl.isIdInterpolated()); + } + @Test void parseIfCondition() { final LALScriptModel model = LALScriptParser.parse( @@ -184,9 +244,286 @@ void parseIfCondition() { assertEquals(2, ifBlock.getThenBranch().size()); } + @Test + void parseElseIfChain() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " if (parsed.a) {\n" + + " sink {}\n" + + " } else if (parsed.b) {\n" + + " sink {}\n" + + " } else if (parsed.c) {\n" + + " sink {}\n" + + " } else {\n" + + " sink {}\n" + + " }\n" + + "}"); + + assertEquals(1, model.getStatements().size()); + final LALScriptModel.IfBlock top = + (LALScriptModel.IfBlock) model.getStatements().get(0); + assertNotNull(top.getCondition()); + assertEquals(1, top.getThenBranch().size()); + + // else branch contains a nested IfBlock for "else if (parsed.b)" + assertEquals(1, top.getElseBranch().size()); + final LALScriptModel.IfBlock elseIf1 = + (LALScriptModel.IfBlock) top.getElseBranch().get(0); + assertNotNull(elseIf1.getCondition()); + assertEquals(1, elseIf1.getThenBranch().size()); + + // nested else branch contains another IfBlock for "else if (parsed.c)" + assertEquals(1, elseIf1.getElseBranch().size()); + final LALScriptModel.IfBlock elseIf2 = + (LALScriptModel.IfBlock) elseIf1.getElseBranch().get(0); + assertNotNull(elseIf2.getCondition()); + assertEquals(1, elseIf2.getThenBranch().size()); + + // innermost else branch is the final else body + assertEquals(1, elseIf2.getElseBranch().size()); + assertInstanceOf(LALScriptModel.SinkBlock.class, elseIf2.getElseBranch().get(0)); + } + + @Test + void parseElseIfWithoutFinalElse() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " if (parsed.a) {\n" + + " sink {}\n" + + " } else if (parsed.b) {\n" + + " sink {}\n" + + " }\n" + + "}"); + + final LALScriptModel.IfBlock top = + (LALScriptModel.IfBlock) model.getStatements().get(0); + assertEquals(1, top.getElseBranch().size()); + final LALScriptModel.IfBlock elseIf = + (LALScriptModel.IfBlock) top.getElseBranch().get(0); + assertNotNull(elseIf.getCondition()); + assertTrue(elseIf.getElseBranch().isEmpty()); + } + @Test void parseSyntaxErrorThrows() { assertThrows(IllegalArgumentException.class, () -> LALScriptParser.parse("filter {")); } + + // ==================== Function call parsing ==================== + + @Test + void parseTagFunctionCallInCondition() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " if (tag(\"LOG_KIND\") == \"SLOW_SQL\") {\n" + + " sink {}\n" + + " }\n" + + "}"); + + final LALScriptModel.IfBlock ifBlock = + (LALScriptModel.IfBlock) model.getStatements().get(0); + final LALScriptModel.ComparisonCondition cond = + (LALScriptModel.ComparisonCondition) ifBlock.getCondition(); + + // Left side should be a function call + final LALScriptModel.ValueAccess left = cond.getLeft(); + assertEquals("tag", left.getFunctionCallName()); + assertEquals(1, left.getFunctionCallArgs().size()); + assertEquals("LOG_KIND", + left.getFunctionCallArgs().get(0).getValue().getSegments().get(0)); + + // Right side should be a string value (parsed as ValueAccess with stringLiteral flag) + assertInstanceOf(LALScriptModel.ValueAccessConditionValue.class, cond.getRight()); + final LALScriptModel.ValueAccessConditionValue rightVal = + (LALScriptModel.ValueAccessConditionValue) cond.getRight(); + assertTrue(rightVal.getValue().isStringLiteral()); + assertEquals("SLOW_SQL", rightVal.getValue().getSegments().get(0)); + } + + @Test + void parseTagFunctionCallAsSingleCondition() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " if (tag(\"LOG_KIND\")) {\n" + + " sink {}\n" + + " }\n" + + "}"); + + final LALScriptModel.IfBlock ifBlock = + (LALScriptModel.IfBlock) model.getStatements().get(0); + final LALScriptModel.ExprCondition cond = + (LALScriptModel.ExprCondition) ifBlock.getCondition(); + assertEquals("tag", cond.getExpr().getFunctionCallName()); + assertEquals(1, cond.getExpr().getFunctionCallArgs().size()); + } + + // ==================== Safe navigation parsing ==================== + + @Test + void parseSafeNavigationFields() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " extractor {\n" + + " service parsed?.response?.service as String\n" + + " }\n" + + " sink {}\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(0); + final LALScriptModel.FieldAssignment field = + (LALScriptModel.FieldAssignment) extractor.getStatements().get(0); + + assertTrue(field.getValue().isParsedRef()); + assertEquals(2, field.getValue().getChain().size()); + assertTrue(((LALScriptModel.FieldSegment) field.getValue().getChain().get(0)) + .isSafeNav()); + assertTrue(((LALScriptModel.FieldSegment) field.getValue().getChain().get(1)) + .isSafeNav()); + } + + @Test + void parseSafeNavigationMethods() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " extractor {\n" + + " service parsed?.flags?.toString()?.trim() as String\n" + + " }\n" + + " sink {}\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(0); + final LALScriptModel.FieldAssignment field = + (LALScriptModel.FieldAssignment) extractor.getStatements().get(0); + + assertEquals(3, field.getValue().getChain().size()); + // flags is a safe field + assertInstanceOf(LALScriptModel.FieldSegment.class, + field.getValue().getChain().get(0)); + assertTrue(((LALScriptModel.FieldSegment) field.getValue().getChain().get(0)) + .isSafeNav()); + // toString() is a safe method + assertInstanceOf(LALScriptModel.MethodSegment.class, + field.getValue().getChain().get(1)); + assertTrue(((LALScriptModel.MethodSegment) field.getValue().getChain().get(1)) + .isSafeNav()); + assertEquals("toString", + ((LALScriptModel.MethodSegment) field.getValue().getChain().get(1)).getName()); + // trim() is a safe method + assertTrue(((LALScriptModel.MethodSegment) field.getValue().getChain().get(2)) + .isSafeNav()); + assertEquals("trim", + ((LALScriptModel.MethodSegment) field.getValue().getChain().get(2)).getName()); + } + + // ==================== Method argument parsing ==================== + + @Test + void parseMethodWithArguments() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " service ProcessRegistry.generateVirtualLocalProcess(" + + "parsed.service as String, parsed.instance as String) as String\n" + + " }\n" + + " sink {}\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(1); + final LALScriptModel.FieldAssignment field = + (LALScriptModel.FieldAssignment) extractor.getStatements().get(0); + + assertTrue(field.getValue().isProcessRegistryRef()); + assertEquals(1, field.getValue().getChain().size()); + + final LALScriptModel.MethodSegment method = + (LALScriptModel.MethodSegment) field.getValue().getChain().get(0); + assertEquals("generateVirtualLocalProcess", method.getName()); + assertEquals(2, method.getArguments().size()); + assertTrue(method.getArguments().get(0).getValue().isParsedRef()); + assertEquals("String", method.getArguments().get(0).getCastType()); + assertTrue(method.getArguments().get(1).getValue().isParsedRef()); + assertEquals("String", method.getArguments().get(1).getCastType()); + } + + // ==================== Sampled trace parsing ==================== + + @Test + void parseSampledTrace() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " sampledTrace {\n" + + " latency parsed.latency as Long\n" + + " uri parsed.uri as String\n" + + " reason parsed.reason as String\n" + + " detectPoint parsed.detect_point as String\n" + + " componentId 49\n" + + " }\n" + + " }\n" + + " sink {}\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(1); + final LALScriptModel.SampledTraceBlock st = + (LALScriptModel.SampledTraceBlock) extractor.getStatements().get(0); + assertEquals(5, st.getStatements().size()); + } + + // ==================== If in extractor/sink parsing ==================== + + @Test + void parseIfInsideExtractor() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " json {}\n" + + " extractor {\n" + + " if (parsed.status) {\n" + + " tag 'http.status_code': parsed.status\n" + + " }\n" + + " tag 'response.flag': parsed.flags\n" + + " }\n" + + " sink {}\n" + + "}"); + + final LALScriptModel.ExtractorBlock extractor = + (LALScriptModel.ExtractorBlock) model.getStatements().get(1); + assertEquals(2, extractor.getStatements().size()); + assertInstanceOf(LALScriptModel.IfBlock.class, extractor.getStatements().get(0)); + assertInstanceOf(LALScriptModel.TagAssignment.class, extractor.getStatements().get(1)); + } + + @Test + void parseIfInsideSink() { + final LALScriptModel model = LALScriptParser.parse( + "filter {\n" + + " sink {\n" + + " sampler {\n" + + " if (parsed.error) {\n" + + " rateLimit('svc:err') {\n" + + " rpm 6000\n" + + " }\n" + + " } else {\n" + + " rateLimit('svc:ok') {\n" + + " rpm 3000\n" + + " }\n" + + " }\n" + + " }\n" + + " }\n" + + "}"); + + final LALScriptModel.SinkBlock sink = + (LALScriptModel.SinkBlock) model.getStatements().get(0); + final LALScriptModel.SamplerBlock sampler = + (LALScriptModel.SamplerBlock) sink.getStatements().get(0); + // The sampler has one if-block as content + assertEquals(1, sampler.getContents().size()); + assertInstanceOf(LALScriptModel.IfBlock.class, sampler.getContents().get(0)); + } } From 4963d2b0cc27089b2d00258fbac90b28236d0da8 Mon Sep 17 00:00:00 2001 From: Wu Sheng Date: Sun, 1 Mar 2026 19:35:21 +0800 Subject: [PATCH 13/64] Restructure test/script-compiler into test/script-cases with shared scripts and runtime comparison - Rename test/script-compiler to test/script-cases/script-runtime-with-groovy - Copy all shipped production configs into test/script-cases/scripts/ as test copies (MAL: test-otel-rules, test-meter-analyzer-config, test-log-mal-rules, test-envoy-metrics-rules; LAL: test-lal; Hierarchy: test-hierarchy-definition.yml) - Update all checker tests to load from shared scripts/ directory - Upgrade LAL checker from compile-only to full runtime execution comparison (v1 Groovy vs v2 ANTLR4+Javassist, comparing Binding state: service, layer, tags, abort/save) - Update Maven coordinates and root pom module path Co-Authored-By: Claude Opus 4.6 --- docs/en/academy/dsl-compiler-design.md | 30 +- pom.xml | 2 +- .../hierarchy-v1-v2-checker/pom.xml | 2 +- .../config/HierarchyRuleComparisonTest.java | 24 +- .../test/resources/hierarchy-definition.yml | 0 .../hierarchy-v1-with-groovy/pom.xml | 2 +- .../config/GroovyHierarchyRuleProvider.java | 0 .../lal-v1-with-groovy/pom.xml | 2 +- .../oap/log/analyzer/dsl/Binding.java | 0 .../skywalking/oap/log/analyzer/dsl/DSL.java | 0 .../analyzer/dsl/LALPrecompiledExtension.java | 0 .../oap/log/analyzer/dsl/LalExpression.java | 0 .../log/analyzer/dsl/spec/AbstractSpec.java | 0 .../dsl/spec/LALDelegatingScript.java | 0 .../dsl/spec/extractor/ExtractorSpec.java | 0 .../sampledtrace/SampledTraceSpec.java | 0 .../spec/extractor/slowsql/SlowSqlSpec.java | 0 .../analyzer/dsl/spec/filter/FilterSpec.java | 0 .../dsl/spec/parser/AbstractParserSpec.java | 0 .../dsl/spec/parser/JsonParserSpec.java | 0 .../dsl/spec/parser/TextParserSpec.java | 0 .../dsl/spec/parser/YamlParserSpec.java | 0 .../analyzer/dsl/spec/sink/SamplerSpec.java | 0 .../log/analyzer/dsl/spec/sink/SinkSpec.java | 0 .../spec/sink/sampler/PossibilitySampler.java | 0 .../sink/sampler/RateLimitingSampler.java | 0 .../dsl/spec/sink/sampler/Sampler.java | 0 .../analyzer/module/LogAnalyzerModule.java | 0 .../oap/log/analyzer/provider/LALConfig.java | 0 .../oap/log/analyzer/provider/LALConfigs.java | 0 .../provider/LogAnalyzerModuleConfig.java | 0 .../provider/LogAnalyzerModuleProvider.java | 0 .../log/ILogAnalysisListenerManager.java | 0 .../provider/log/ILogAnalyzerService.java | 0 .../analyzer/provider/log/LogAnalyzer.java | 0 .../provider/log/LogAnalyzerServiceImpl.java | 0 .../log/analyzer/LogAnalyzerFactory.java | 0 .../log/listener/LogAnalysisListener.java | 0 .../listener/LogAnalysisListenerFactory.java | 0 .../log/listener/LogFilterListener.java | 0 .../log/listener/LogSinkListener.java | 0 .../log/listener/LogSinkListenerFactory.java | 0 .../log/listener/RecordSinkListener.java | 0 .../log/listener/TrafficSinkListener.java | 0 ...ing.oap.server.library.module.ModuleDefine | 0 ...g.oap.server.library.module.ModuleProvider | 0 .../oap/log/analyzer/dsl/DSLSecurityTest.java | 0 .../oap/log/analyzer/dsl/DSLTest.java | 0 .../resources/log-mal-rules/placeholder.yaml | 0 .../org.mockito.plugins.MockMaker | 0 .../mal-lal-v1-v2-checker/pom.xml | 2 +- .../oap/server/checker/InMemoryCompiler.java | 0 .../server/checker/lal/LalComparisonTest.java | 346 ++++++++++++++++++ .../server/checker/mal/MalComparisonTest.java | 32 +- .../checker/mal/MalFilterComparisonTest.java | 30 +- .../mal-v1-with-groovy/pom.xml | 2 +- .../oap/meter/analyzer/Analyzer.java | 0 .../oap/meter/analyzer/MetricConvert.java | 0 .../oap/meter/analyzer/MetricRuleConfig.java | 0 .../oap/meter/analyzer/dsl/DSL.java | 0 .../meter/analyzer/dsl/DownsamplingType.java | 0 .../EndpointEntityDescription.java | 0 .../EntityDescription/EntityDescription.java | 0 .../InstanceEntityDescription.java | 0 .../ProcessEntityDescription.java | 0 .../ProcessRelationEntityDescription.java | 0 .../ServiceEntityDescription.java | 0 .../ServiceRelationEntityDescription.java | 0 .../oap/meter/analyzer/dsl/Expression.java | 0 .../dsl/ExpressionParsingContext.java | 0 .../dsl/ExpressionParsingException.java | 0 .../meter/analyzer/dsl/FilterExpression.java | 0 .../oap/meter/analyzer/dsl/MalExpression.java | 0 .../oap/meter/analyzer/dsl/MalFilter.java | 0 .../oap/meter/analyzer/dsl/NumberClosure.java | 0 .../oap/meter/analyzer/dsl/Result.java | 0 .../oap/meter/analyzer/dsl/Sample.java | 0 .../oap/meter/analyzer/dsl/SampleFamily.java | 0 .../analyzer/dsl/SampleFamilyBuilder.java | 0 .../analyzer/dsl/SampleFamilyFunctions.java | 0 .../analyzer/dsl/counter/CounterWindow.java | 0 .../oap/meter/analyzer/dsl/counter/ID.java | 0 .../dsl/registry/ProcessRegistry.java | 0 .../analyzer/dsl/tagOpt/K8sRetagType.java | 0 .../oap/meter/analyzer/dsl/tagOpt/Retag.java | 0 .../meter/analyzer/k8s/K8sInfoRegistry.java | 0 .../prometheus/PrometheusMetricConverter.java | 0 .../analyzer/prometheus/rule/MetricsRule.java | 0 .../meter/analyzer/prometheus/rule/Rule.java | 0 .../meter/analyzer/prometheus/rule/Rules.java | 0 .../oap/meter/analyzer/MetricConvertTest.java | 0 .../meter/analyzer/dsl/AggregationTest.java | 0 .../oap/meter/analyzer/dsl/AnalyzerTest.java | 0 .../meter/analyzer/dsl/ArithmeticTest.java | 0 .../oap/meter/analyzer/dsl/BasicTest.java | 0 .../oap/meter/analyzer/dsl/DecorateTest.java | 0 .../analyzer/dsl/ExpressionParsingTest.java | 0 .../oap/meter/analyzer/dsl/FilterTest.java | 0 .../oap/meter/analyzer/dsl/FunctionTest.java | 0 .../oap/meter/analyzer/dsl/IncreaseTest.java | 0 .../oap/meter/analyzer/dsl/K8sTagTest.java | 0 .../oap/meter/analyzer/dsl/ScopeTest.java | 0 .../oap/meter/analyzer/dsl/TagFilterTest.java | 0 .../meter/analyzer/dsl/ValueFilterTest.java | 0 .../dsl/counter/CounterWindowTest.java | 0 .../analyzer/dsl/rule/RuleLoaderFailTest.java | 0 .../analyzer/dsl/rule/RuleLoaderTest.java | 0 .../dsl/rule/RuleLoaderYAMLFailTest.java | 0 .../org.mockito.plugins.MockMaker | 0 .../otel-rules/illegal-yaml/test.yml | 0 .../otel-rules/single-file-case.yaml | 0 .../otel-rules/test-folder/case1.yaml | 0 .../otel-rules/test-folder/case2.yml | 0 .../otel-rules/test-folder/case3.yaml | 0 .../test-folder/deeperFolder/caseUnReach.yaml | 0 .../otel-rules/test-folder/empty.yaml | 0 .../script-runtime-with-groovy}/pom.xml | 4 +- .../test-hierarchy-definition.yml | 123 +++++++ .../scripts/lal/test-lal/default.yaml | 24 ++ .../scripts/lal/test-lal/envoy-als.yaml | 92 +++++ .../scripts/lal/test-lal/k8s-service.yaml | 61 +++ .../scripts/lal/test-lal/mesh-dp.yaml | 60 +++ .../scripts/lal/test-lal/mysql-slowsql.yaml | 35 ++ .../scripts/lal/test-lal/nginx.yaml | 60 +++ .../scripts/lal/test-lal/pgsql-slowsql.yaml | 35 ++ .../scripts/lal/test-lal/redis-slowsql.yaml | 35 ++ .../envoy-svc-relation.yaml | 48 +++ .../mal/test-envoy-metrics-rules/envoy.yaml | 77 ++++ .../scripts/mal/test-log-mal-rules/nginx.yaml | 36 ++ .../mal/test-log-mal-rules/placeholder.yaml | 16 + .../continuous-profiling.yaml | 28 ++ .../datasource.yaml | 20 + .../test-meter-analyzer-config/go-agent.yaml | 32 ++ .../go-runtime.yaml | 80 ++++ .../java-agent.yaml | 32 ++ .../network-profiling.yaml | 135 +++++++ .../python-runtime.yaml | 36 ++ .../ruby-runtime.yaml | 53 +++ .../test-meter-analyzer-config/satellite.yaml | 34 ++ .../spring-micrometer.yaml | 64 ++++ .../threadpool.yaml | 20 + .../activemq/activemq-broker.yaml | 97 +++++ .../activemq/activemq-cluster.yaml | 95 +++++ .../activemq/activemq-destination.yaml | 79 ++++ .../scripts/mal/test-otel-rules/apisix.yaml | 102 ++++++ .../aws-dynamodb/dynamodb-endpoint.yaml | 85 +++++ .../aws-dynamodb/dynamodb-service.yaml | 108 ++++++ .../test-otel-rules/aws-eks/eks-cluster.yaml | 52 +++ .../mal/test-otel-rules/aws-eks/eks-node.yaml | 69 ++++ .../test-otel-rules/aws-eks/eks-service.yaml | 48 +++ .../aws-gateway/gateway-endpoint.yaml | 63 ++++ .../aws-gateway/gateway-service.yaml | 63 ++++ .../test-otel-rules/aws-s3/s3-service.yaml | 55 +++ .../banyandb/banyandb-instance.yaml | 86 +++++ .../banyandb/banyandb-service.yaml | 86 +++++ .../bookkeeper/bookkeeper-cluster.yaml | 62 ++++ .../bookkeeper/bookkeeper-node.yaml | 90 +++++ .../clickhouse/clickhouse-instance.yaml | 178 +++++++++ .../clickhouse/clickhouse-service.yaml | 162 ++++++++ .../elasticsearch/elasticsearch-cluster.yaml | 72 ++++ .../elasticsearch/elasticsearch-index.yaml | 127 +++++++ .../elasticsearch/elasticsearch-node.yaml | 149 ++++++++ .../mal/test-otel-rules/flink/flink-job.yaml | 72 ++++ .../flink/flink-jobManager.yaml | 80 ++++ .../flink/flink-taskManager.yaml | 89 +++++ .../test-otel-rules/istio-controlplane.yaml | 108 ++++++ .../mal/test-otel-rules/k8s/k8s-cluster.yaml | 94 +++++ .../mal/test-otel-rules/k8s/k8s-instance.yaml | 23 ++ .../mal/test-otel-rules/k8s/k8s-node.yaml | 74 ++++ .../mal/test-otel-rules/k8s/k8s-service.yaml | 59 +++ .../test-otel-rules/kafka/kafka-broker.yaml | 118 ++++++ .../test-otel-rules/kafka/kafka-cluster.yaml | 58 +++ .../test-otel-rules/kong/kong-endpoint.yaml | 40 ++ .../test-otel-rules/kong/kong-instance.yaml | 105 ++++++ .../test-otel-rules/kong/kong-service.yaml | 69 ++++ .../mongodb/mongodb-cluster.yaml | 63 ++++ .../test-otel-rules/mongodb/mongodb-node.yaml | 108 ++++++ .../test-otel-rules/mysql/mysql-instance.yaml | 82 +++++ .../test-otel-rules/mysql/mysql-service.yaml | 74 ++++ .../test-otel-rules/nginx/nginx-endpoint.yaml | 49 +++ .../test-otel-rules/nginx/nginx-instance.yaml | 51 +++ .../test-otel-rules/nginx/nginx-service.yaml | 51 +++ .../scripts/mal/test-otel-rules/oap.yaml | 144 ++++++++ .../postgresql/postgresql-instance.yaml | 115 ++++++ .../postgresql/postgresql-service.yaml | 95 +++++ .../test-otel-rules/pulsar/pulsar-broker.yaml | 76 ++++ .../pulsar/pulsar-cluster.yaml | 68 ++++ .../rabbitmq/rabbitmq-cluster.yaml | 86 +++++ .../rabbitmq/rabbitmq-node.yaml | 80 ++++ .../test-otel-rules/redis/redis-instance.yaml | 69 ++++ .../test-otel-rules/redis/redis-service.yaml | 70 ++++ .../rocketmq/rocketmq-broker.yaml | 46 +++ .../rocketmq/rocketmq-cluster.yaml | 81 ++++ .../rocketmq/rocketmq-topic.yaml | 70 ++++ .../scripts/mal/test-otel-rules/vm.yaml | 97 +++++ .../scripts/mal/test-otel-rules/windows.yaml | 72 ++++ .../server/checker/lal/LalComparisonTest.java | 153 -------- 197 files changed, 6222 insertions(+), 209 deletions(-) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/hierarchy-v1-v2-checker/pom.xml (97%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java (91%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/hierarchy-v1-with-groovy/pom.xml (96%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/pom.xml (96%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-lal-v1-v2-checker/pom.xml (98%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java (100%) create mode 100644 test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java (90%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java (89%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/pom.xml (96%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml (100%) rename test/{script-compiler => script-cases/script-runtime-with-groovy}/pom.xml (92%) create mode 100644 test/script-cases/scripts/hierarchy-rule/test-hierarchy-definition.yml create mode 100644 test/script-cases/scripts/lal/test-lal/default.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/envoy-als.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/k8s-service.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/mesh-dp.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/mysql-slowsql.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/nginx.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/pgsql-slowsql.yaml create mode 100644 test/script-cases/scripts/lal/test-lal/redis-slowsql.yaml create mode 100644 test/script-cases/scripts/mal/test-envoy-metrics-rules/envoy-svc-relation.yaml create mode 100644 test/script-cases/scripts/mal/test-envoy-metrics-rules/envoy.yaml create mode 100644 test/script-cases/scripts/mal/test-log-mal-rules/nginx.yaml create mode 100644 test/script-cases/scripts/mal/test-log-mal-rules/placeholder.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/continuous-profiling.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/datasource.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/go-agent.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/go-runtime.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/java-agent.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/network-profiling.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/python-runtime.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/ruby-runtime.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/satellite.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/spring-micrometer.yaml create mode 100644 test/script-cases/scripts/mal/test-meter-analyzer-config/threadpool.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/activemq/activemq-broker.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/activemq/activemq-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/activemq/activemq-destination.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/apisix.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-dynamodb/dynamodb-endpoint.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-dynamodb/dynamodb-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-eks/eks-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-eks/eks-node.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-eks/eks-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-gateway/gateway-endpoint.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-gateway/gateway-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/aws-s3/s3-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/banyandb/banyandb-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/banyandb/banyandb-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/bookkeeper/bookkeeper-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/bookkeeper/bookkeeper-node.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/clickhouse/clickhouse-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/clickhouse/clickhouse-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/elasticsearch/elasticsearch-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/elasticsearch/elasticsearch-index.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/elasticsearch/elasticsearch-node.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/flink/flink-job.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/flink/flink-jobManager.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/flink/flink-taskManager.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/istio-controlplane.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/k8s/k8s-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/k8s/k8s-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/k8s/k8s-node.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/k8s/k8s-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/kafka/kafka-broker.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/kafka/kafka-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/kong/kong-endpoint.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/kong/kong-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/kong/kong-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/mongodb/mongodb-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/mongodb/mongodb-node.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/mysql/mysql-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/mysql/mysql-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/nginx/nginx-endpoint.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/nginx/nginx-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/nginx/nginx-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/oap.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/postgresql/postgresql-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/postgresql/postgresql-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/pulsar/pulsar-broker.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/pulsar/pulsar-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/rabbitmq/rabbitmq-node.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/redis/redis-instance.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/redis/redis-service.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/rocketmq/rocketmq-broker.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/rocketmq/rocketmq-cluster.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/rocketmq/rocketmq-topic.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/vm.yaml create mode 100644 test/script-cases/scripts/mal/test-otel-rules/windows.yaml delete mode 100644 test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java diff --git a/docs/en/academy/dsl-compiler-design.md b/docs/en/academy/dsl-compiler-design.md index ce14082e2084..7e243754b8d6 100644 --- a/docs/en/academy/dsl-compiler-design.md +++ b/docs/en/academy/dsl-compiler-design.md @@ -123,25 +123,31 @@ work exactly as before. Only the internal compilation engine was replaced. ### Verification: Groovy v1 Checker To ensure the new Java compilers produce identical results to the original Groovy implementation, -a **dual-path comparison test suite** is maintained under `test/script-compiler/`: +a **dual-path comparison test suite** is maintained under `test/script-cases/`: ``` -test/script-compiler/ - mal-groovy/ # MAL v1: original Groovy-based implementation - lal-groovy/ # LAL v1: original Groovy-based implementation - hierarchy-groovy/ # Hierarchy v1: original Groovy-based implementation - mal-lal-v1-v2-checker/ # Runs every MAL/LAL expression through BOTH v1 and v2, compares results - hierarchy-v1-v2-checker/ # Runs every hierarchy rule through BOTH v1 and v2, compares results +test/script-cases/ + scripts/ + mal/ # Copies of shipped MAL configs (test-otel-rules, test-meter-analyzer-config, etc.) + lal/ # Copies of shipped LAL scripts (test-lal/) + hierarchy-rule/ # Copy of shipped hierarchy-definition.yml + script-runtime-with-groovy/ + mal-v1-with-groovy/ # MAL v1: original Groovy-based implementation + lal-v1-with-groovy/ # LAL v1: original Groovy-based implementation + hierarchy-v1-with-groovy/ # Hierarchy v1: original Groovy-based implementation + mal-lal-v1-v2-checker/ # Runs every MAL/LAL expression through BOTH v1 and v2, compares results + hierarchy-v1-v2-checker/ # Runs every hierarchy rule through BOTH v1 and v2, compares results ``` The checker mechanism: -1. Loads all production YAML config files (the same files used by OAP at runtime) +1. Loads all test copies of production YAML config files from `test/script-cases/scripts/` 2. For each DSL expression, compiles with **both** v1 (Groovy) and v2 (ANTLR4 + Javassist) -3. Compares the compiled artifacts: - - **MAL**: Compare generated Java source code, extracted metadata (sample names, aggregation labels, - downsampling type, percentile config), and execution results with sample data - - **LAL**: Compare compiled expression execution against FilterSpec/Binding mocks +3. Compares the results: + - **MAL**: Compare extracted metadata (sample names, aggregation labels, + downsampling type, percentile config) + - **LAL**: Runtime execution comparison — both v1 and v2 execute with mock LogData, + then compare Binding state (service, layer, tags, abort/save flags) - **Hierarchy**: Compare `BiFunction` evaluation with test Service pairs This ensures 100% behavioral parity. The Groovy v1 modules are **test-only dependencies** -- they are not diff --git a/pom.xml b/pom.xml index 9beacb8b7797..c517f685d2f2 100755 --- a/pom.xml +++ b/pom.xml @@ -83,7 +83,7 @@ oap-server oap-server-bom - test/script-compiler + test/script-cases/script-runtime-with-groovy diff --git a/test/script-compiler/hierarchy-v1-v2-checker/pom.xml b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/pom.xml similarity index 97% rename from test/script-compiler/hierarchy-v1-v2-checker/pom.xml rename to test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/pom.xml index 1c753e0f2f45..8ecbf8d55e4f 100644 --- a/test/script-compiler/hierarchy-v1-v2-checker/pom.xml +++ b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/pom.xml @@ -19,7 +19,7 @@ - script-compiler + script-runtime-with-groovy org.apache.skywalking ${revision} diff --git a/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java similarity index 91% rename from test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java rename to test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java index c859bf1e6d55..2a661de0c180 100644 --- a/test/script-compiler/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java +++ b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/core/config/HierarchyRuleComparisonTest.java @@ -17,15 +17,16 @@ package org.apache.skywalking.oap.server.core.config; -import java.io.FileNotFoundException; +import java.io.FileReader; import java.io.Reader; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.ArrayList; import java.util.Collection; import java.util.List; import java.util.Map; import java.util.function.BiFunction; import org.apache.skywalking.oap.server.core.query.type.Service; -import org.apache.skywalking.oap.server.library.util.ResourceUtils; import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import org.apache.skywalking.oap.server.core.config.compiler.CompiledHierarchyRuleProvider; @@ -64,8 +65,8 @@ private static class TestPair { @SuppressWarnings("unchecked") @TestFactory - Collection allRulesProduceIdenticalResults() throws FileNotFoundException { - final Reader reader = ResourceUtils.read("hierarchy-definition.yml"); + Collection allRulesProduceIdenticalResults() throws Exception { + final Reader reader = new FileReader(findHierarchyDefinition().toFile()); final Yaml yaml = new Yaml(); final Map config = yaml.loadAs(reader, Map.class); final Map ruleExpressions = (Map) config.get("auto-matching-rules"); @@ -186,4 +187,19 @@ private static List testPairsFor(final String ruleName) { } return pairs; } + + private Path findHierarchyDefinition() { + final String[] candidates = { + "test/script-cases/scripts/hierarchy-rule/test-hierarchy-definition.yml", + "../../scripts/hierarchy-rule/test-hierarchy-definition.yml" + }; + for (final String candidate : candidates) { + final Path path = Path.of(candidate); + if (Files.isRegularFile(path)) { + return path; + } + } + throw new IllegalStateException( + "Cannot find test-hierarchy-definition.yml in scripts/hierarchy-rule/"); + } } diff --git a/test/script-compiler/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml similarity index 100% rename from test/script-compiler/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml rename to test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker/src/test/resources/hierarchy-definition.yml diff --git a/test/script-compiler/hierarchy-v1-with-groovy/pom.xml b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-with-groovy/pom.xml similarity index 96% rename from test/script-compiler/hierarchy-v1-with-groovy/pom.xml rename to test/script-cases/script-runtime-with-groovy/hierarchy-v1-with-groovy/pom.xml index d4f039cb3aff..c5efa459d62f 100644 --- a/test/script-compiler/hierarchy-v1-with-groovy/pom.xml +++ b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-with-groovy/pom.xml @@ -19,7 +19,7 @@ - script-compiler + script-runtime-with-groovy org.apache.skywalking ${revision} diff --git a/test/script-compiler/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java b/test/script-cases/script-runtime-with-groovy/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java similarity index 100% rename from test/script-compiler/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java rename to test/script-cases/script-runtime-with-groovy/hierarchy-v1-with-groovy/src/main/java/org/apache/skywalking/oap/server/core/config/GroovyHierarchyRuleProvider.java diff --git a/test/script-compiler/lal-v1-with-groovy/pom.xml b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/pom.xml similarity index 96% rename from test/script-compiler/lal-v1-with-groovy/pom.xml rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/pom.xml index adae2c5b03bc..fe98943a8c0b 100644 --- a/test/script-compiler/lal-v1-with-groovy/pom.xml +++ b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/pom.xml @@ -18,7 +18,7 @@ - script-compiler + script-runtime-with-groovy org.apache.skywalking ${revision} diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/Binding.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/DSL.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LALPrecompiledExtension.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/LalExpression.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/AbstractSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/LALDelegatingScript.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/ExtractorSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/sampledtrace/SampledTraceSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/extractor/slowsql/SlowSqlSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/filter/FilterSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/AbstractParserSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/JsonParserSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/TextParserSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/parser/YamlParserSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SamplerSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/SinkSpec.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/PossibilitySampler.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/RateLimitingSampler.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/dsl/spec/sink/sampler/Sampler.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/module/LogAnalyzerModule.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfig.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LALConfigs.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleConfig.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/LogAnalyzerModuleProvider.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalysisListenerManager.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/ILogAnalyzerService.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzer.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/LogAnalyzerServiceImpl.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/analyzer/LogAnalyzerFactory.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListener.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogAnalysisListenerFactory.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogFilterListener.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListener.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/LogSinkListenerFactory.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/RecordSinkListener.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/log/analyzer/provider/log/listener/TrafficSinkListener.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleDefine diff --git a/test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/main/resources/META-INF/services/org.apache.skywalking.oap.server.library.module.ModuleProvider diff --git a/test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLSecurityTest.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/log/analyzer/dsl/DSLTest.java diff --git a/test/script-compiler/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/resources/log-mal-rules/placeholder.yaml diff --git a/test/script-compiler/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker similarity index 100% rename from test/script-compiler/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker rename to test/script-cases/script-runtime-with-groovy/lal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker diff --git a/test/script-compiler/mal-lal-v1-v2-checker/pom.xml b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/pom.xml similarity index 98% rename from test/script-compiler/mal-lal-v1-v2-checker/pom.xml rename to test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/pom.xml index d4e036f0a807..9233b0828ae1 100644 --- a/test/script-compiler/mal-lal-v1-v2-checker/pom.xml +++ b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/pom.xml @@ -19,7 +19,7 @@ - script-compiler + script-runtime-with-groovy org.apache.skywalking ${revision} diff --git a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java similarity index 100% rename from test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java rename to test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/InMemoryCompiler.java diff --git a/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java new file mode 100644 index 000000000000..e69c2185acfa --- /dev/null +++ b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/lal/LalComparisonTest.java @@ -0,0 +1,346 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.checker.lal; + +import java.io.File; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.apache.skywalking.apm.network.common.v3.KeyStringValuePair; +import org.apache.skywalking.apm.network.logging.v3.JSONLog; +import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.apm.network.logging.v3.LogDataBody; +import org.apache.skywalking.apm.network.logging.v3.LogTags; +import org.apache.skywalking.oap.log.analyzer.compiler.LALClassGenerator; +import org.apache.skywalking.oap.log.analyzer.dsl.Binding; +import org.apache.skywalking.oap.log.analyzer.dsl.DSL; +import org.apache.skywalking.oap.log.analyzer.dsl.spec.filter.FilterSpec; +import org.apache.skywalking.oap.log.analyzer.module.LogAnalyzerModule; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleConfig; +import org.apache.skywalking.oap.log.analyzer.provider.LogAnalyzerModuleProvider; +import org.apache.skywalking.oap.server.core.CoreModule; +import org.apache.skywalking.oap.server.core.config.ConfigService; +import org.apache.skywalking.oap.server.core.source.SourceReceiver; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder; +import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder; +import org.junit.jupiter.api.DynamicTest; +import org.junit.jupiter.api.TestFactory; +import org.yaml.snakeyaml.Yaml; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.fail; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +/** + * Dual-path comparison test for LAL (Log Analysis Language) scripts. + * For each LAL rule across all LAL YAML files: + *
    + *
  • Path A (v1): Groovy compilation + runtime execution via {@link DSL}
    • + *
  • Path B (v2): ANTLR4 + Javassist compilation via {@link LALClassGenerator} + * + runtime execution via reflective {@code execute(Object, Object)} call
    • + *
+ * Both paths are fed the same mock LogData and the resulting Binding state + * (service, layer, tags, abort/save flags) is compared. + */ +class LalComparisonTest { + + @TestFactory + Collection lalScriptsCompileAndExecute() throws Exception { + final List tests = new ArrayList<>(); + final Map> yamlRules = loadAllLalYamlFiles(); + + for (final Map.Entry> entry : yamlRules.entrySet()) { + final String yamlFile = entry.getKey(); + for (final LalRule rule : entry.getValue()) { + tests.add(DynamicTest.dynamicTest( + yamlFile + " | " + rule.name, + () -> compareExecution(rule.name, rule.dsl) + )); + } + } + + return tests; + } + + private void compareExecution(final String ruleName, + final String dsl) throws Exception { + final ModuleManager manager = buildMockModuleManager(); + final LogData testLog = buildTestLogData(dsl); + + // ---- V1: Groovy path (via DSL.of which uses GroovyShell internally) ---- + Binding v1Binding = null; + try { + final DSL v1Dsl = DSL.of(manager, new LogAnalyzerModuleConfig(), dsl); + disableSinkListeners(v1Dsl); + + v1Binding = new Binding().log(testLog); + v1Dsl.bind(v1Binding); + v1Dsl.evaluate(); + } catch (Exception e) { + // V1 failed — skip comparison + } + + // ---- V2: ANTLR4 + Javassist compilation + execution ---- + Binding v2Binding = null; + String v2Error = null; + try { + final LALClassGenerator generator = new LALClassGenerator(); + final Object v2Expr = generator.compile(dsl); + + final FilterSpec v2FilterSpec = new FilterSpec(manager, new LogAnalyzerModuleConfig()); + disableSinkListenersOnSpec(v2FilterSpec); + + v2Binding = new Binding().log(testLog); + v2FilterSpec.bind(v2Binding); + + // Call execute(Object, Object) via reflection since the Javassist-generated + // class declares execute(Object, Object) rather than the typed signature + final Method executeMethod = v2Expr.getClass().getMethod("execute", + Object.class, Object.class); + executeMethod.invoke(v2Expr, v2FilterSpec, v2Binding); + } catch (Exception e) { + final Throwable cause = e.getCause() != null ? e.getCause() : e; + v2Error = cause.getClass().getSimpleName() + ": " + cause.getMessage(); + } + + // ---- Compare ---- + if (v1Binding == null && v2Binding == null) { + return; + } + if (v1Binding == null) { + // V1 failed but v2 succeeded — v2 is more capable, OK + return; + } + if (v2Binding == null) { + fail(ruleName + ": v2 execution failed but v1 succeeded — " + v2Error); + return; + } + + // Compare binding state + assertEquals(v1Binding.shouldAbort(), v2Binding.shouldAbort(), + ruleName + ": shouldAbort mismatch"); + assertEquals(v1Binding.shouldSave(), v2Binding.shouldSave(), + ruleName + ": shouldSave mismatch"); + + final LogData.Builder v1Log = v1Binding.log(); + final LogData.Builder v2Log = v2Binding.log(); + + assertEquals(v1Log.getService(), v2Log.getService(), + ruleName + ": service mismatch"); + assertEquals(v1Log.getServiceInstance(), v2Log.getServiceInstance(), + ruleName + ": serviceInstance mismatch"); + assertEquals(v1Log.getEndpoint(), v2Log.getEndpoint(), + ruleName + ": endpoint mismatch"); + assertEquals(v1Log.getLayer(), v2Log.getLayer(), + ruleName + ": layer mismatch"); + assertEquals(v1Log.getTimestamp(), v2Log.getTimestamp(), + ruleName + ": timestamp mismatch"); + assertEquals(v1Log.getTags(), v2Log.getTags(), + ruleName + ": tags mismatch"); + } + + private ModuleManager buildMockModuleManager() { + final ModuleManager manager = mock(ModuleManager.class); + setInternalField(manager, "isInPrepareStage", false); + when(manager.find(anyString())).thenReturn(mock(ModuleProviderHolder.class)); + + final ModuleProviderHolder logAnalyzerHolder = mock(ModuleProviderHolder.class); + final LogAnalyzerModuleProvider logAnalyzerProvider = mock(LogAnalyzerModuleProvider.class); + when(logAnalyzerProvider.getMetricConverts()).thenReturn(Collections.emptyList()); + when(logAnalyzerHolder.provider()).thenReturn(logAnalyzerProvider); + when(manager.find(LogAnalyzerModule.NAME)).thenReturn(logAnalyzerHolder); + + when(manager.find(CoreModule.NAME).provider()).thenReturn(mock(ModuleServiceHolder.class)); + when(manager.find(CoreModule.NAME).provider().getService(SourceReceiver.class)) + .thenReturn(mock(SourceReceiver.class)); + when(manager.find(CoreModule.NAME).provider().getService(ConfigService.class)) + .thenReturn(mock(ConfigService.class)); + when(manager.find(CoreModule.NAME) + .provider() + .getService(ConfigService.class) + .getSearchableLogsTags()) + .thenReturn(""); + return manager; + } + + private LogData buildTestLogData(final String dsl) { + final LogData.Builder builder = LogData.newBuilder() + .setService("test-service") + .setServiceInstance("test-instance") + .setTimestamp(System.currentTimeMillis()); + + if (dsl.contains("json")) { + builder.setBody(LogDataBody.newBuilder() + .setJson(JSONLog.newBuilder() + .setJson("{\"level\":\"ERROR\",\"msg\":\"test\"," + + "\"layer\":\"GENERAL\",\"service\":\"test-svc\"," + + "\"time\":\"1234567890\"," + + "\"id\":\"slow-1\",\"statement\":\"SELECT 1\"," + + "\"query_time\":500}"))); + } + + if (dsl.contains("LOG_KIND")) { + builder.setTags(LogTags.newBuilder() + .addData(KeyStringValuePair.newBuilder() + .setKey("LOG_KIND").setValue("SLOW_SQL"))); + } + + return builder.build(); + } + + private void disableSinkListeners(final Object dsl) { + try { + final Object filterSpec = getInternalField(dsl, "filterSpec"); + setInternalField(filterSpec, "sinkListenerFactories", Collections.emptyList()); + } catch (Exception e) { + // Best effort + } + } + + private void disableSinkListenersOnSpec(final FilterSpec filterSpec) { + try { + setInternalField(filterSpec, "sinkListenerFactories", Collections.emptyList()); + } catch (Exception e) { + // Best effort + } + } + + private static void setInternalField(final Object target, final String fieldName, + final Object value) { + try { + Field field = null; + Class clazz = target.getClass(); + while (clazz != null && field == null) { + try { + field = clazz.getDeclaredField(fieldName); + } catch (NoSuchFieldException e) { + clazz = clazz.getSuperclass(); + } + } + if (field != null) { + field.setAccessible(true); + field.set(target, value); + } + } catch (Exception e) { + // ignore + } + } + + private static Object getInternalField(final Object target, final String fieldName) { + try { + Field field = null; + Class clazz = target.getClass(); + while (clazz != null && field == null) { + try { + field = clazz.getDeclaredField(fieldName); + } catch (NoSuchFieldException e) { + clazz = clazz.getSuperclass(); + } + } + if (field != null) { + field.setAccessible(true); + return field.get(target); + } + } catch (Exception e) { + // ignore + } + return null; + } + + @SuppressWarnings("unchecked") + private Map> loadAllLalYamlFiles() throws Exception { + final Map> result = new HashMap<>(); + final Yaml yaml = new Yaml(); + + final Path scriptsDir = findScriptsDir("lal"); + if (scriptsDir == null) { + return result; + } + final Path lalDir = scriptsDir.resolve("test-lal"); + if (!Files.isDirectory(lalDir)) { + return result; + } + + final File[] files = lalDir.toFile().listFiles(); + if (files == null) { + return result; + } + for (final File file : files) { + if (!file.getName().endsWith(".yaml") && !file.getName().endsWith(".yml")) { + continue; + } + final String content = Files.readString(file.toPath()); + final Map config = yaml.load(content); + if (config == null || !config.containsKey("rules")) { + continue; + } + final List> rules = + (List>) config.get("rules"); + if (rules == null) { + continue; + } + final List lalRules = new ArrayList<>(); + for (final Map rule : rules) { + final String name = rule.get("name"); + final String dslStr = rule.get("dsl"); + if (name == null || dslStr == null) { + continue; + } + lalRules.add(new LalRule(name, dslStr)); + } + if (!lalRules.isEmpty()) { + result.put("lal/" + file.getName(), lalRules); + } + } + return result; + } + + private Path findScriptsDir(final String language) { + final String[] candidates = { + "test/script-cases/scripts/" + language, + "../../scripts/" + language + }; + for (final String candidate : candidates) { + final Path path = Path.of(candidate); + if (Files.isDirectory(path)) { + return path; + } + } + return null; + } + + private static class LalRule { + final String name; + final String dsl; + + LalRule(final String name, final String dsl) { + this.name = name; + this.dsl = dsl; + } + } +} diff --git a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java similarity index 90% rename from test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java rename to test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java index 3b6117217523..6784e3852a58 100644 --- a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java +++ b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalComparisonTest.java @@ -25,7 +25,6 @@ import java.util.HashMap; import java.util.List; import java.util.Map; -import java.util.concurrent.atomic.AtomicInteger; import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.oap.meter.analyzer.compiler.MALClassGenerator; import org.apache.skywalking.oap.meter.analyzer.dsl.DSL; @@ -38,6 +37,7 @@ import org.yaml.snakeyaml.Yaml; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.fail; /** * Dual-path comparison test for MAL (Meter Analysis Language) expressions. @@ -52,8 +52,6 @@ @Slf4j class MalComparisonTest { - private static final AtomicInteger V2_COMPILE_GAPS = new AtomicInteger(); - @TestFactory Collection malExpressionsMatch() throws Exception { final List tests = new ArrayList<>(); @@ -96,15 +94,15 @@ private void compareExpression(final String metricName, // ---- Compare ---- if (v1Ctx == null && v2Meta == null) { + // Both failed — consistent behavior return; } if (v1Ctx == null) { + // V1 failed but v2 succeeded — v2 is more capable, OK return; } if (v2Meta == null) { - V2_COMPILE_GAPS.incrementAndGet(); - log.info("V2 compile gap for '{}': {}", metricName, v2Error); - return; + fail(metricName + ": v2 compile failed but v1 succeeded — " + v2Error); } // Both succeeded - compare metadata @@ -128,16 +126,18 @@ private Map> loadAllMalYamlFiles() throws Exception { final Yaml yaml = new Yaml(); final String[] dirs = { - "meter-analyzer-config", - "otel-rules" + "test-meter-analyzer-config", + "test-otel-rules" }; - for (final String dir : dirs) { - final Path dirPath = findResourceDir(dir); - if (dirPath == null) { - continue; + final Path scriptsDir = findScriptsDir("mal"); + if (scriptsDir != null) { + for (final String dir : dirs) { + final Path dirPath = scriptsDir.resolve(dir); + if (Files.isDirectory(dirPath)) { + collectYamlFiles(dirPath.toFile(), dir, yaml, result); + } } - collectYamlFiles(dirPath.toFile(), dir, yaml, result); } return result; @@ -197,10 +197,10 @@ private void collectYamlFiles(final File dir, final String prefix, } } - private Path findResourceDir(final String name) { + private Path findScriptsDir(final String language) { final String[] candidates = { - "oap-server/server-starter/src/main/resources/" + name, - "../../../oap-server/server-starter/src/main/resources/" + name + "test/script-cases/scripts/" + language, + "../../scripts/" + language }; for (final String candidate : candidates) { final Path path = Path.of(candidate); diff --git a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java similarity index 89% rename from test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java rename to test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java index 4400701691ca..4c7d121f6096 100644 --- a/test/script-compiler/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java +++ b/test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker/src/test/java/org/apache/skywalking/oap/server/checker/mal/MalFilterComparisonTest.java @@ -142,19 +142,17 @@ private Set collectAllFilterExpressions() throws Exception { final Set filters = new LinkedHashSet<>(); final Yaml yaml = new Yaml(); - final String[] dirs = {"meter-analyzer-config", "otel-rules"}; - for (final String dir : dirs) { - final Path dirPath = findResourceDir(dir); - if (dirPath == null) { - continue; - } - collectFiltersFromDir(dirPath.toFile(), yaml, filters); - } - - for (final String dir : new String[]{"log-mal-rules", "envoy-metrics-rules"}) { - final Path dirPath = findResourceDir(dir); - if (dirPath != null) { - collectFiltersFromDir(dirPath.toFile(), yaml, filters); + final String[] dirs = { + "test-meter-analyzer-config", "test-otel-rules", + "test-log-mal-rules", "test-envoy-metrics-rules" + }; + final Path scriptsDir = findScriptsDir("mal"); + if (scriptsDir != null) { + for (final String dir : dirs) { + final Path dirPath = scriptsDir.resolve(dir); + if (Files.isDirectory(dirPath)) { + collectFiltersFromDir(dirPath.toFile(), yaml, filters); + } } } @@ -191,10 +189,10 @@ private void collectFiltersFromDir(final File dir, final Yaml yaml, } } - private Path findResourceDir(final String name) { + private Path findScriptsDir(final String language) { final String[] candidates = { - "oap-server/server-starter/src/main/resources/" + name, - "../../../oap-server/server-starter/src/main/resources/" + name + "test/script-cases/scripts/" + language, + "../../scripts/" + language }; for (final String candidate : candidates) { final Path path = Path.of(candidate); diff --git a/test/script-compiler/mal-v1-with-groovy/pom.xml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/pom.xml similarity index 96% rename from test/script-compiler/mal-v1-with-groovy/pom.xml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/pom.xml index 0fa3ad21065e..2abc0aa17480 100644 --- a/test/script-compiler/mal-v1-with-groovy/pom.xml +++ b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/pom.xml @@ -19,7 +19,7 @@ - script-compiler + script-runtime-with-groovy org.apache.skywalking ${revision} diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricConvert.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/MetricRuleConfig.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DSL.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/DownsamplingType.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EndpointEntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/EntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/InstanceEntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessEntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ProcessRelationEntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceEntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/EntityDescription/ServiceRelationEntityDescription.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Expression.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingContext.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingException.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterExpression.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalExpression.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/MalFilter.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/NumberClosure.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Result.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/Sample.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamily.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyBuilder.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/SampleFamilyFunctions.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindow.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/ID.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/registry/ProcessRegistry.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/K8sRetagType.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/dsl/tagOpt/Retag.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/k8s/K8sInfoRegistry.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/PrometheusMetricConverter.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/MetricsRule.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rule.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/main/java/org/apache/skywalking/oap/meter/analyzer/prometheus/rule/Rules.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/MetricConvertTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AggregationTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/AnalyzerTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ArithmeticTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/BasicTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/DecorateTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ExpressionParsingTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FilterTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/FunctionTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/IncreaseTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/K8sTagTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ScopeTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/TagFilterTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/ValueFilterTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/counter/CounterWindowTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderFailTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/java/org/apache/skywalking/oap/meter/analyzer/dsl/rule/RuleLoaderYAMLFailTest.java diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/illegal-yaml/test.yml diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/single-file-case.yaml diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case1.yaml diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case2.yml diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/case3.yaml diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/deeperFolder/caseUnReach.yaml diff --git a/test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml b/test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml similarity index 100% rename from test/script-compiler/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml rename to test/script-cases/script-runtime-with-groovy/mal-v1-with-groovy/src/test/resources/otel-rules/test-folder/empty.yaml diff --git a/test/script-compiler/pom.xml b/test/script-cases/script-runtime-with-groovy/pom.xml similarity index 92% rename from test/script-compiler/pom.xml rename to test/script-cases/script-runtime-with-groovy/pom.xml index 24d1865c4783..caafca02b679 100644 --- a/test/script-compiler/pom.xml +++ b/test/script-cases/script-runtime-with-groovy/pom.xml @@ -22,11 +22,11 @@ oap-server org.apache.skywalking ${revision} - ../../oap-server/pom.xml + ../../../oap-server/pom.xml 4.0.0 - script-compiler + script-runtime-with-groovy pom diff --git a/test/script-cases/scripts/hierarchy-rule/test-hierarchy-definition.yml b/test/script-cases/scripts/hierarchy-rule/test-hierarchy-definition.yml new file mode 100644 index 000000000000..1f44cf5630b3 --- /dev/null +++ b/test/script-cases/scripts/hierarchy-rule/test-hierarchy-definition.yml @@ -0,0 +1,123 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Define the hierarchy of service layers, the layers under the specific layer are related lower of the layer. +# The relation could have a matching rule for auto matching, which are defined in the `auto-matching-rules` section. +# All the layers are defined in the file `org.apache.skywalking.oap.server.core.analysis.Layers.java`. +# Notice: some hierarchy relations and auto matching rules are only works on k8s env. + +hierarchy: + MESH: + MESH_DP: name + K8S_SERVICE: short-name + + MESH_DP: + K8S_SERVICE: short-name + + GENERAL: + APISIX: lower-short-name-remove-ns + K8S_SERVICE: lower-short-name-remove-ns + KONG: lower-short-name-remove-ns + + MYSQL: + K8S_SERVICE: short-name + + POSTGRESQL: + K8S_SERVICE: short-name + + APISIX: + K8S_SERVICE: short-name + + NGINX: + K8S_SERVICE: short-name + + SO11Y_OAP: + K8S_SERVICE: short-name + + ROCKETMQ: + K8S_SERVICE: short-name + + RABBITMQ: + K8S_SERVICE: short-name + + KAFKA: + K8S_SERVICE: short-name + + CLICKHOUSE: + K8S_SERVICE: short-name + + PULSAR: + K8S_SERVICE: short-name + + ACTIVEMQ: + K8S_SERVICE: short-name + + KONG: + K8S_SERVICE: short-name + + VIRTUAL_DATABASE: + MYSQL: lower-short-name-with-fqdn + POSTGRESQL: lower-short-name-with-fqdn + CLICKHOUSE: lower-short-name-with-fqdn + + VIRTUAL_MQ: + ROCKETMQ: lower-short-name-with-fqdn + RABBITMQ: lower-short-name-with-fqdn + KAFKA: lower-short-name-with-fqdn + PULSAR: lower-short-name-with-fqdn + + CILIUM_SERVICE: + K8S_SERVICE: short-name + +# Use Groovy script to define the matching rules, the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean, +# which are used to match the relation between the upper service(u) and the lower service(l) on the different layers. +auto-matching-rules: + # the name of the upper service is equal to the name of the lower service + name: "{ (u, l) -> u.name == l.name }" + # the short name of the upper service is equal to the short name of the lower service + short-name: "{ (u, l) -> u.shortName == l.shortName }" + # remove the k8s namespace from the lower service short name + # this rule is only works on k8s env. + lower-short-name-remove-ns: "{ (u, l) -> { if(l.shortName.lastIndexOf('.') > 0) return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); return false; } }" + # the short name of the upper remove port is equal to the short name of the lower service with fqdn suffix + # this rule is only works on k8s env. + lower-short-name-with-fqdn: "{ (u, l) -> { if(u.shortName.lastIndexOf(':') > 0) return u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local'); return false; } }" + +# The hierarchy level of the service layer, the level is used to define the order of the service layer for UI presentation. +# The level of the upper service should greater than the level of the lower service in `hierarchy` section. +layer-levels: + MESH: 3 + GENERAL: 3 + SO11Y_OAP: 3 + VIRTUAL_DATABASE: 3 + VIRTUAL_MQ: 3 + + MYSQL: 2 + POSTGRESQL: 2 + APISIX: 2 + NGINX: 2 + ROCKETMQ: 2 + CLICKHOUSE: 2 + RABBITMQ: 2 + KAFKA: 2 + PULSAR: 2 + ACTIVEMQ: 2 + KONG: 2 + + MESH_DP: 1 + CILIUM_SERVICE: 1 + + K8S_SERVICE: 0 + diff --git a/test/script-cases/scripts/lal/test-lal/default.yaml b/test/script-cases/scripts/lal/test-lal/default.yaml new file mode 100644 index 000000000000..12317a95bf55 --- /dev/null +++ b/test/script-cases/scripts/lal/test-lal/default.yaml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The default LAL script to save all logs, behaving like the versions before 8.5.0. +rules: + - name: default + layer: GENERAL + dsl: | + filter { + sink { + } + } diff --git a/test/script-cases/scripts/lal/test-lal/envoy-als.yaml b/test/script-cases/scripts/lal/test-lal/envoy-als.yaml new file mode 100644 index 000000000000..e6530c3fa713 --- /dev/null +++ b/test/script-cases/scripts/lal/test-lal/envoy-als.yaml @@ -0,0 +1,92 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +rules: + - name: envoy-als + layer: MESH + dsl: | + filter { + // only collect abnormal logs (http status code >= 300, or commonProperties?.responseFlags is not empty) + if (parsed?.response?.responseCode?.value as Integer < 400 && !parsed?.commonProperties?.responseFlags?.toString()?.trim()) { + abort {} + } + extractor { + if (parsed?.response?.responseCode) { + tag 'status.code': parsed?.response?.responseCode?.value + } + tag 'response.flag': parsed?.commonProperties?.responseFlags + } + sink { + sampler { + if (parsed?.commonProperties?.responseFlags?.toString()) { + // use service:errorCode as sampler id so that each service:errorCode has its own sampler, + // e.g. checkoutservice:[upstreamConnectionFailure], checkoutservice:[upstreamRetryLimitExceeded] + rateLimit("${log.service}:${parsed?.commonProperties?.responseFlags?.toString()}") { + rpm 6000 + } + } else { + // use service:responseCode as sampler id so that each service:responseCode has its own sampler, + // e.g. checkoutservice:500, checkoutservice:404. + rateLimit("${log.service}:${parsed?.response?.responseCode}") { + rpm 6000 + } + } + } + } + } + - name: network-profiling-slow-trace + layer: MESH + dsl: | + filter { + json{ + } + extractor{ + if (tag("LOG_KIND") == "NET_PROFILING_SAMPLED_TRACE") { + sampledTrace { + latency parsed.latency as Long + uri parsed.uri as String + reason parsed.reason as String + + if (parsed.client_process.process_id as String != "") { + processId parsed.client_process.process_id as String + } else if (parsed.client_process.local as Boolean) { + processId ProcessRegistry.generateVirtualLocalProcess(parsed.service as String, parsed.serviceInstance as String) as String + } else { + processId ProcessRegistry.generateVirtualRemoteProcess(parsed.service as String, parsed.serviceInstance as String, parsed.client_process.address as String) as String + } + + if (parsed.server_process.process_id as String != "") { + destProcessId parsed.server_process.process_id as String + } else if (parsed.server_process.local as Boolean) { + destProcessId ProcessRegistry.generateVirtualLocalProcess(parsed.service as String, parsed.serviceInstance as String) as String + } else { + destProcessId ProcessRegistry.generateVirtualRemoteProcess(parsed.service as String, parsed.serviceInstance as String, parsed.server_process.address as String) as String + } + + detectPoint parsed.detect_point as String + + if (parsed.component as String == "http" && parsed.ssl as Boolean) { + componentId 129 + } else if (parsed.component as String == "http") { + componentId 49 + } else if (parsed.ssl as Boolean) { + componentId 130 + } else { + componentId 110 + } + } + } + } + } diff --git a/test/script-cases/scripts/lal/test-lal/k8s-service.yaml b/test/script-cases/scripts/lal/test-lal/k8s-service.yaml new file mode 100644 index 000000000000..2992b39ed7a7 --- /dev/null +++ b/test/script-cases/scripts/lal/test-lal/k8s-service.yaml @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The default LAL script to save all logs, behaving like the versions before 8.5.0. +rules: + - name: network-profiling-slow-trace + layer: K8S_SERVICE + dsl: | + filter { + json{ + } + extractor{ + if (tag("LOG_KIND") == "NET_PROFILING_SAMPLED_TRACE") { + sampledTrace { + latency parsed.latency as Long + uri parsed.uri as String + reason parsed.reason as String + + if (parsed.client_process.process_id as String != "") { + processId parsed.client_process.process_id as String + } else if (parsed.client_process.local as Boolean) { + processId ProcessRegistry.generateVirtualLocalProcess(parsed.service as String, parsed.serviceInstance as String) as String + } else { + processId ProcessRegistry.generateVirtualRemoteProcess(parsed.service as String, parsed.serviceInstance as String, parsed.client_process.address as String) as String + } + + if (parsed.server_process.process_id as String != "") { + destProcessId parsed.server_process.process_id as String + } else if (parsed.server_process.local as Boolean) { + destProcessId ProcessRegistry.generateVirtualLocalProcess(parsed.service as String, parsed.serviceInstance as String) as String + } else { + destProcessId ProcessRegistry.generateVirtualRemoteProcess(parsed.service as String, parsed.serviceInstance as String, parsed.server_process.address as String) as String + } + + detectPoint parsed.detect_point as String + + if (parsed.component as String == "http" && parsed.ssl as Boolean) { + componentId 129 + } else if (parsed.component as String == "http") { + componentId 49 + } else if (parsed.ssl as Boolean) { + componentId 130 + } else { + componentId 110 + } + } + } + } + } \ No newline at end of file diff --git a/test/script-cases/scripts/lal/test-lal/mesh-dp.yaml b/test/script-cases/scripts/lal/test-lal/mesh-dp.yaml new file mode 100644 index 000000000000..e8271ef8a708 --- /dev/null +++ b/test/script-cases/scripts/lal/test-lal/mesh-dp.yaml @@ -0,0 +1,60 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +rules: + - name: network-profiling-slow-trace + layer: MESH_DP + dsl: | + filter { + json{ + } + extractor{ + if (tag("LOG_KIND") == "NET_PROFILING_SAMPLED_TRACE") { + sampledTrace { + latency parsed.latency as Long + uri parsed.uri as String + reason parsed.reason as String + + if (parsed.client_process.process_id as String != "") { + processId parsed.client_process.process_id as String + } else if (parsed.client_process.local as Boolean) { + processId ProcessRegistry.generateVirtualLocalProcess(parsed.service as String, parsed.serviceInstance as String) as String + } else { + processId ProcessRegistry.generateVirtualRemoteProcess(parsed.service as String, parsed.serviceInstance as String, parsed.client_process.address as String) as String + } + + if (parsed.server_process.process_id as String != "") { + destProcessId parsed.server_process.process_id as String + } else if (parsed.server_process.local as Boolean) { + destProcessId ProcessRegistry.generateVirtualLocalProcess(parsed.service as String, parsed.serviceInstance as String) as String + } else { + destProcessId ProcessRegistry.generateVirtualRemoteProcess(parsed.service as String, parsed.serviceInstance as String, parsed.server_process.address as String) as String + } + + detectPoint parsed.detect_point as String + + if (parsed.component as String == "http" && parsed.ssl as Boolean) { + componentId 129 + } else if (parsed.component as String == "http") { + componentId 49 + } else if (parsed.ssl as Boolean) { + componentId 130 + } else { + componentId 110 + } + } + } + } + } \ No newline at end of file diff --git a/test/script-cases/scripts/lal/test-lal/mysql-slowsql.yaml b/test/script-cases/scripts/lal/test-lal/mysql-slowsql.yaml new file mode 100644 index 000000000000..774da2955db6 --- /dev/null +++ b/test/script-cases/scripts/lal/test-lal/mysql-slowsql.yaml @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +rules: + - name: mysql-slowsql + layer: MYSQL + dsl: | + filter { + json{ + } + extractor{ + layer parsed.layer as String + service parsed.service as String + timestamp parsed.time as String + if (tag("LOG_KIND") == "SLOW_SQL") { + slowSql { + id parsed.id as String + statement parsed.statement as String + latency parsed.query_time as Long + } + } + } + } diff --git a/test/script-cases/scripts/lal/test-lal/nginx.yaml b/test/script-cases/scripts/lal/test-lal/nginx.yaml new file mode 100644 index 000000000000..d6c50dd4c0fd --- /dev/null +++ b/test/script-cases/scripts/lal/test-lal/nginx.yaml @@ -0,0 +1,60 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +rules: + - name: nginx-access-log + layer: NGINX + dsl: | + filter { + if (tag("LOG_KIND") == "NGINX_ACCESS_LOG") { + text { + regexp $/.+ \"(?.+)\" (?\d{3}) .+/$ + } + + extractor { + if (parsed.status) { + tag 'http.status_code': parsed.status + } + } + + sink { + } + } + } + - name: nginx-error-log + layer: NGINX + dsl: | + filter { + if (tag("LOG_KIND") == "NGINX_ERROR_LOG") { + text { + regexp $/(?