Working with Data

Learn how to load, process, and transform data in Noodles.gl for your visualizations.

Data Sources

Use a FileOp to read a file from a URL or text. Supported formats:

Format	Output Type	Use Case
json	Parsed object/array	JSON data files
csv	Array of row objects	Tabular data
text	Raw string	Plain text files, custom parsing
binary	ArrayBuffer/Uint8Array	Images, binary files

Common Data Formats

JSON Data

{
  "flights": [
    {
      "origin": "SFO",
      "destination": "LAX",
      "coordinates": [-122.4194, 37.7749]
    }
  ]
}

CSV Data

origin,destination,passengers,coordinates
SFO,LAX,150,"[-122.4194, 37.7749]"

GeoJSON

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [-122.4194, 37.7749]
      },
      "properties": {
        "name": "SFO",
        "passengers": 25000
      }
    }
  ]
}

Text Format

Use text format to load raw file contents as a string, useful for custom parsing:

// FileOp with format: 'text'
// Output is the raw file contents as a string

// Then in a CodeOp, parse however you need:
const lines = data.split('\n')
const parsed = lines.map(line => {
  const [name, value] = line.split(':')
  return { name: name.trim(), value: parseFloat(value) }
})
return parsed

Binary Format

Use binary format for non-text files like images or custom binary data:

// FileOp with format: 'binary'
// Output is ArrayBuffer (from URL) or Uint8Array (from text input)

// Example: Process binary data in a CodeOp
const buffer = data  // ArrayBuffer from FileOp
const view = new DataView(buffer)
const header = view.getUint32(0, true)  // Read 4-byte header

Data Processing

Code Operators

Run custom JavaScript code on the data. Use data to access the input data list, d for the first element of the list, and op to access other operators. Also passes utils like turf and d3, and all Noodles.gl operator classes as variables. Use this to store state.

// CodeOp
const csvData = d // input data list connected to FileOp
const parsed = d3.csvParse(csvData) // parse CSV string using available d3 utility
const filtered = parsed.filter(row => row.passengers > 100)
return turf.featureCollection(filtered.map(row => (
  turf.point([row.lng, row.lat], { name: row.name, passengers: row.passengers })
)))

Referencing Operators

Use the op function to reference data from other operators.

// In a CodeOp, reference other nodes:
op('/data-source/my-csv').out.data       // Absolute path
op('./sibling-operator').out.val         // Relative path
op('../parent-container/node').out.val   // Parent navigation

Free Exports

Use utilities from turf, d3, or import any ESM module.

// CodeOp - use a pre-bundled d3 utility
return d3.scaleLinear()
  .domain([0, 100])
  .range([0, 1])

// CodeOp - import an ESM module
const _ = await import('https://esm.sh/lodash')
return _.mapValues(
  _.groupBy(d, 'name'),
  group => _.sortBy(group, 'ts'),
)

// CodeOp - use turf for geospatial calculations
const buffered = d.map(feature =>
  turf.buffer(feature, 5, { units: 'kilometers' })
)
return buffered

Common Data Tasks

Coordinate Conversion

// Convert lat/lng to [lng, lat] for Deck.gl as a reactive reference
const data = op('./raw-coordinates').out.data
return data.map(point => ({
  ...point,
  coordinates: [point.longitude, point.latitude] // [lng, lat]
}))

Color Mapping

// Map data values to colors
const data = op('./flight-data').out.data
const maxPassengers = Math.max(...data.map(d => d.passengers))

return data.map(flight => ({
  ...flight,
  color: [
    255 * (flight.passengers / maxPassengers), // Red
    0,                                         // Green
    255 * (1 - flight.passengers / maxPassengers) // Blue
  ]
}))

Time-based Filtering

// Filter by time range for animations
const allData = op('./complete-dataset').out.data // FileOp
const currentTime = op('./current-time').out.val // NumberOp

return allData.filter(item => {
  const itemTime = new Date(item.timestamp).getTime()
  return itemTime <= currentTime
})

Performance Tips

Large Datasets

• Use aggregation operators before visualization
• Filter early in your pipeline
• Consider sampling for real-time preview
• Use efficient data structures (arrays vs objects)

Memory Management

• Avoid copying large datasets unnecessarily
• Use references instead of duplicating data
• Clean up intermediate calculations
• Monitor browser memory usage

Debugging Data Issues

Data Preview

• Use ViewerOp and ConsoleOp to inspect intermediate results
• Check data types and structure
• Verify coordinate formats ([lng, lat])

Common Issues

• Wrong coordinate order: Use [longitude, latitude]
• String vs number: Ensure numeric fields are parsed
• Missing data: Check for null/undefined handling
• Array structure: Deck.gl expects arrays, not objects