Error Analysis & Debugging

Learn how to analyze errors, debug failed runs, and improve your pipeline reliability.

Understanding Error Types

Runtime Errors

Errors that occur during script execution, such as syntax errors, import failures, or unhandled exceptions.

Status: failed

Error: ModuleNotFoundError: No module named 'pandas'

Validation Errors

Data quality issues detected by your validation checks.

Status: failed

Error: Data validation failed - 45 null values found

Timeout Errors

Script exceeded maximum execution time or heartbeat missed.

Status: failed

Error: Heartbeat timeout - No signal received for 15 minutes

Error Analysis Dashboard

Viewing Error Details

  1. Navigate to the Jobs page in your dashboard
  2. Click on a job with failed status
  3. View the Run History section for detailed error logs
  4. Click on a specific failed run to see full error traceback

Error Metadata

Each error includes detailed metadata to help with debugging:

  • Exit code (non-zero indicates failure)
  • Error message and stack trace
  • Execution duration before failure
  • Timestamp of failure
  • Custom metadata from your script
Error Patterns & Trends

Identifying Recurring Issues

Use the dashboard analytics to spot patterns:

  • Failure Rate: Track the percentage of runs that fail over time
  • Error Frequency: See which errors occur most often
  • Time-Based Patterns: Identify if failures happen at specific times
  • Environment Issues: Check if failures correlate with system resources

Common Error Scenarios

Memory Issues

Large datasets causing out-of-memory errors

Solution: Process data in chunks or increase system resources

External API Failures

Third-party service timeouts or rate limits

Solution: Implement retry logic and exponential backoff

Data Quality Issues

Unexpected data formats or missing values

Solution: Add validation checks and handle edge cases

Debugging Best Practices

1. Add Detailed Logging

import logging
from seerpy import Seer

seer = Seer(api_key="your_api_key")
logging.basicConfig(level=logging.INFO)

try:
    seer.log("Starting data processing...")
    process_data()
    seer.log("Data processing completed successfully")
except Exception as e:
    logging.error(f"Error processing data: {str(e)}")
    seer.log(f"ERROR: {str(e)}", {"traceback": traceback.format_exc()})
    raise

2. Use Metadata for Context

seer.start(
    job_name="data-pipeline",
    metadata={
        "environment": "production",
        "data_source": "database_a",
        "row_count": len(df),
        "columns": list(df.columns)
    }
)

3. Implement Checkpoints

Add progress markers to identify where failures occur:

seer.log("Checkpoint 1: Data loaded")
seer.log("Checkpoint 2: Validation passed")
seer.log("Checkpoint 3: Transformation complete")
seer.log("Checkpoint 4: Data saved")
Related Resources