Skip to content

Python Jobs Automation#

The LIT Platform's Python Jobs system allows you to automate LLM workflows and scripts on customizable schedules. This powerful feature enables you to build applications that run periodically without human intervention, perfect for data processing, report generation, and ongoing monitoring tasks.

Features#

  • Scheduled Execution: Run workflows on precise schedules
  • Event-Based Triggers: Trigger jobs based on system events
  • Monitoring and Alerts: Get notifications about job status
  • Job History: View detailed execution history and logs
  • Resource Management: Control CPU, memory, and GPU allocation
  • Dependencies: Manage job dependencies and execution order
  • Error Handling: Implement retry strategies and fallbacks

Job Types#

The Python Jobs system supports several types of automated tasks:

LLM Workflows#

Run any workflow created in the LLM Workflow Canvas on a schedule:

  1. Create a workflow in the LLM Studio
  2. Navigate to Python Jobs > New Job
  3. Select "LLM Workflow" as the job type
  4. Choose your workflow from the dropdown
  5. Configure execution schedule
  6. Set input parameters

Python Scripts#

Execute custom Python scripts with full access to the LIT Platform's capabilities:

  1. Navigate to Python Jobs > New Job
  2. Select "Python Script" as the job type
  3. Write or upload your Python script
  4. Import LIT modules as needed (import lit.llm, etc.)
  5. Configure execution schedule
  6. Set environment variables and parameters

Data Processing#

Automate data processing tasks:

  1. Navigate to Python Jobs > New Job
  2. Select "Data Processing" as the job type
  3. Configure data sources and destinations
  4. Define processing steps
  5. Set execution schedule

Scheduling Options#

Time-Based Scheduling#

Set jobs to run at specific times using cron syntax:

  • Every hour: 0 * * * *
  • Daily at midnight: 0 0 * * *
  • Every Monday at 9am: 0 9 * * 1
  • First day of each month: 0 0 1 * *
  • Every 15 minutes: */15 * * * *

The scheduling interface provides both a visual builder and direct cron syntax input.

Event-Based Triggers#

Trigger jobs based on system events:

  • File uploads
  • Database changes
  • API calls
  • Workflow completions
  • System alerts

Manual Triggers#

Jobs can also be triggered manually:

  • From the Jobs dashboard
  • Through the REST API
  • Via command line interface

Job Configuration#

Input Parameters#

Jobs can accept parameters from various sources:

  • Fixed values defined at configuration time
  • Environment variables
  • Output from other jobs
  • File contents
  • Database queries
  • API responses

Resource Allocation#

Control the resources available to each job:

  • CPU cores
  • Memory limits
  • GPU allocation
  • Maximum runtime
  • Disk space

Notifications#

Configure notifications for job status:

  • Email alerts
  • Slack messages
  • Webhook calls
  • Dashboard notifications

Monitoring and Management#

Jobs Dashboard#

The Jobs Dashboard provides a comprehensive view of all automated tasks:

  • Current status of each job
  • Next scheduled execution time
  • Last execution result
  • Resource utilization
  • Success/failure statistics

Detailed Job Logs#

Each job execution generates detailed logs:

  • Execution start and end times
  • Input parameters
  • Output values
  • Error messages
  • Resource usage statistics
  • Execution trace

Job Controls#

From the dashboard, you can:

  • Start jobs immediately
  • Pause scheduled executions
  • Modify job parameters
  • Clone existing jobs
  • Export job configurations

Example Use Cases#

Daily Report Generation#

Create a job that:

  1. Connects to your database
  2. Extracts yesterday's statistics
  3. Sends the data to an LLM workflow
  4. Generates a natural language summary
  5. Emails the report to stakeholders

Continuous Learning#

Set up a job to:

  1. Monitor a feedback collection endpoint
  2. Periodically process new feedback
  3. Update a vector database with new information
  4. Retrain embeddings if necessary
  5. Update relevant LLM workflows

Content Monitoring#

Create a job that:

  1. Checks specified websites for updates
  2. Extracts and processes new content
  3. Analyzes the content using LLMs
  4. Generates alerts for important changes
  5. Updates an internal knowledge base

Best Practices#

  • Start Simple: Begin with basic jobs and add complexity incrementally
  • Logging: Include detailed logging for troubleshooting
  • Error Handling: Implement robust error handling and recovery
  • Resource Constraints: Set appropriate resource limits
  • Monitoring: Regularly review job performance and results
  • Dependencies: Carefully manage dependencies between jobs
  • Testing: Test jobs thoroughly before scheduling them

The Python Jobs system turns the LIT Platform into a powerful automation engine, enabling you to build sophisticated applications that run continuously without human intervention.