Skip to main content

Creating Operators

Operators are the core processing units in the Noodles.gl system. They take inputs, process data, and produce outputs that can be connected to other operators.

Operator Fundamentals​

Basic Structure​

export class SliceOp extends Operator<SliceOp> {
  static displayName = 'Slice'
  static description = 'Slice an array of data'
  createInputs() {
    return {
      data: new DataField(),
      start: new NumberField(0, { min: 0, step: 1 }),
      end: new NumberField(10, { min: 0, step: 1, optional: true }),
    }
  }
  createOutputs() {
    return {
      data: new DataField(),
    }
  }
  execute({
    data,
    start,
    end,
  }: ExtractProps<typeof this.inputs>): ExtractProps<typeof this.outputs> {
    return {
      data: data.slice(start, end)
    }
  }
}

Key Principles​

  • Pure Functions: Operators should be deterministic (same inputs = same outputs)
  • Reactive: Automatically re-execute when upstream data changes
  • Typed: Use Zod schemas for input/output validation
  • Memoized: Results are cached to avoid unnecessary recomputation

Operator Categories​

Data Operators​

  • FileOp: Load JSON, CSV, text, or binary files from URL or text
  • JSONOp: Parse JSON from text with templating support
  • DuckDbOp: SQL queries with reactive references
  • GeocoderOp: Convert addresses to coordinates

Processing Operators​

  • FilterOp: Filter data based on conditions
  • SliceOp: Select a subset of array elements
  • SortOp: Sort data by field
  • MergeOp: Combine multiple datasets
  • ConcatOp: Concatenate arrays

Math Operators​

  • NumberOp: Numeric constants and calculations
  • ExpressionOp: Single-line JavaScript expressions
  • CodeOp: Multi-line custom JavaScript code
  • AccessorOp: Data accessor functions for Deck.gl

Visualization Operators​

  • ScatterplotLayerOp: Point visualizations
  • PathLayerOp: Line and route visualizations
  • H3HexagonLayerOp: Hexagonal grid visualizations
  • HeatmapLayerOp: Density visualizations

Code Operators​

CodeOp​

For complex data processing with full JavaScript support:

// Example: Calculate distance between points
const distances = data.map(d => {
  const from = [d.start_lng, d.start_lat]
  const to = [d.end_lng, d.end_lat]
  return turf.distance(from, to, { units: 'kilometers' })
})

return distances

Available globals:

  • d3 - D3.js library
  • turf - Turf.js geospatial functions
  • deck - Deck.gl utilities
  • Plot - Observable Plot
  • utils - Utility functions (color conversion, geospatial helpers, KML conversion, etc.)
  • All Operator classes for instantiation

AccessorOp​

For Deck.gl layer accessors (per-item functions):

// Example: Get position from data item
[d.longitude, d.latitude]

// Example: Get color based on value
d.value > 100 ? [255, 0, 0] : [0, 255, 0]

Context:

  • d - Current data item
  • data - Full dataset array

ExpressionOp​

For simple single-line calculations:

// Example: Calculate area
Math.PI * Math.pow(d.radius, 2)

Operator Paths and References​

Fully Qualified Paths​

Operators are identified by Unix-style paths that reflect their position in the container hierarchy:

  • Root operators: /operatorName (e.g., /code1, /threshold)
  • Nested operators: /container/operatorName (e.g., /analysis/filter1)
  • Deep nesting: /container/subcontainer/operatorName

Reactive References​

Use op('path') with fully qualified or relative paths to reference other operators:

// Absolute path reference
const threshold = op('/threshold').par.value
const filtered = data.filter(d => d.value > threshold)

// Relative path reference (same container)
const localData = op('./data-source').out.value
const processed = op('processor').out.value  // equivalent to './processor'

// Parent container reference
const parentConfig = op('../config').par.value

Path Resolution​

The op() function supports Unix-style path resolution, which means you can use ./, ../, and operator to reference other operators relative to the current operator:

  • Absolute paths: /path/to/operator - from root
  • Relative paths: ./operator - same container
  • Parent paths: ../operator - parent container
  • Simple names: operator - same container (equivalent to ./operator)

DuckDB Integration​

DuckDbOp supports SQL with reactive mustache syntax using fully qualified paths:

SELECT * FROM data
WHERE population > {{/config/threshold.par.value}}
ORDER BY population DESC
LIMIT {{limit.par.value}}

Enhanced Mustache Syntax​

The getFieldReferences() function parses mustache templates with path support:

-- Absolute path references
SELECT * FROM data WHERE value > {{/threshold.par.value}}

-- Relative path references (same container)
SELECT * FROM data WHERE value > {{./threshold.par.value}}
SELECT * FROM data WHERE value > {{threshold.par.value}}  -- equivalent

-- Parent container references
SELECT * FROM data WHERE value > {{../config/threshold.par.value}}

-- Complex nested paths
SELECT * FROM data WHERE value > {{../../global/config.par.threshold}}

Automatic Reference Detection:

  • Supports both {{mustache}} and op('path') function syntax
  • Creates reactive connections automatically
  • Resolves paths based on calling operator's context
  • Handles multiple references in single templates

Path resolution in SQL:

  • Absolute paths: {{/container/operator.field.value}}
  • Relative paths: {{./operator.field.value}} or {{operator.field.value}}
  • Parent paths: {{../operator.field.value}}

Best Practices​

Performance​

  • Keep operators pure and stateless
  • Avoid heavy computations in AccessorOps
  • Use memoization for expensive calculations
  • Batch data operations when possible

Data Flow​

  • Design clear input/output contracts
  • Use descriptive field names
  • Validate data at operator boundaries
  • Handle edge cases gracefully

Code Organization​

  • Break complex logic into smaller operators
  • Reuse common patterns as operator templates
  • Document custom operators thoroughly
  • Test operators in isolation

Containers and Hierarchy​

Container Organization​

Operators can be organized into containers to create logical groupings and avoid naming conflicts:

/                         (root)
├── data-loader           (root-level operator)
├── threshold             (root-level operator)
└── analysis/             (container)
    ├── filter1           (nested operator: /analysis/filter1)
    ├── processor         (nested operator: /analysis/processor)
    └── visualization/    (nested container)
        └── map-layer     (deeply nested: /analysis/visualization/map-layer)

Path Benefits​

  • No ID Conflicts: Multiple operators can have the same base name in different containers
  • Logical Organization: Group related operators together
  • Clear References: Explicit paths show data flow relationships
  • Scalability: Support for complex, hierarchical projects

Container Operations​

  • Moving Operators: Drag operators between containers to reorganize
  • Path Updates: References automatically update when operators move
  • Nested Containers: Create containers within containers for complex organization

ForLoop Patterns​

ForLoop operators enable map-like iteration over arrays, where each iteration's result is collected into an output array.

Basic Structure​

[Input Array] → ForLoopBegin → [Processing] → ForLoopEnd → [Output Array]

ForLoopBeginOp outputs:

  • item - Current array element being processed
  • index - Current iteration index (0-based)
  • total - Total number of elements in the array

ForLoopEndOp:

  • Input: item - The result to collect from this iteration
  • Output: data - Array containing ALL collected results

Example: Transform Array Elements​

To double each number in an array [1, 2, 3]:

FileOp (data: [1, 2, 3])
    ↓
ForLoopBegin
    ↓ item (outputs: 1, then 2, then 3)
ExpressionOp (code: `item * 2`)
    ↓
ForLoopEnd (item input)
    ↓ data
Result: [2, 4, 6]

When to Use ForLoop vs CodeOp​

Use ForLoop when:

  • Each iteration needs multiple operators (complex transformations)
  • You want visual debugging of each step
  • The transformation involves other graph nodes

Use CodeOp when:

  • Simple Array.map() suffices
  • All logic fits in one code block
  • Performance is critical (ForLoop has per-iteration overhead)

Custom Operators​

Create new operators by extending the base class:

export class CustomOperator extends Operator<CustomOperator> {
  static displayName = 'Custom Processor'
  static description = 'Processes data with custom logic'

  createInputs() {
    return {
      input: new DataField(),
      threshold: new NumberField(50, { min: 0, max: 100 })
    }
  }

  createOutputs() {
    return {
      result: new DataField()
    }
  }

  execute({ input, threshold }: ExtractProps<typeof this.inputs>): ExtractProps<typeof this.outputs> {
    return {
      result: input.filter(item => item.value > threshold)
    }
  }
}