Set up Google Cloud Logging as an ingest method

Step 1 - Create a Cloud Logging ingest token

Why? Ingest tokens authenticate and authorize data flows from Google Cloud to send data to your repository. They control which parsers can be used and what fields can be populated from Cloud Logging data.

Detailed steps:

Sign in to Falcon LogScale, and browse to your repository.
Click Settings, Ingest Tokens.
Click Add token.
Type in a descriptive name (for example, gcp-cloud-logging-production).
Set the appropriate permissions:
- Assign parser to allow automatic parser selection based on GCP log types
- Assign fields to enable field creation from Cloud Logging metadata and log entries
Click Create token to save the token and securely store the generated string - you'll need this when configuring the Cloud Function or ingestion pipeline.
Note your LogScale ingestion endpoint URL:
- For cloud deployments: Typically https://cloud.humio.com or your regional endpoint
- For on-premises deployments: Your self-hosted LogScale URL
- The full ingestion endpoint will be: https://cloud.humio.com/api/v1/ingest/humio-structured

Step 2 - Plan your Cloud Logging integration

Why? Planning your integration strategy ensures you capture the right logs, optimize costs, and implement the most appropriate architecture for your GCP environment. Understanding Cloud Logging's structure and routing capabilities helps you design an efficient ingestion pipeline.

Detailed steps:

Identify your integration architecture:
- Pub/Sub + Cloud Function: Recommended approach for most use cases
  - Best for: Real-time log streaming, serverless architecture, automatic scaling
  - Advantages: No infrastructure management, automatic scaling, built-in retry logic
  - Considerations: Cloud Function execution costs, cold start latency
  - Flow: Cloud Logging → Log Router → Pub/Sub Topic → Cloud Function → LogScale
- Pub/Sub + Dataflow: For high-volume, complex processing requirements
  - Best for: Very high log volumes (millions of events per second), complex transformations
  - Advantages: Horizontal scaling, sophisticated processing, exactly-once semantics
  - Considerations: Higher complexity, Dataflow job management, increased costs
  - Flow: Cloud Logging → Log Router → Pub/Sub Topic → Dataflow Job → LogScale
- Pub/Sub + Custom Consumer: For specialized processing requirements
  - Best for: Custom processing logic, integration with existing systems
  - Advantages: Full control over processing, custom transformation logic
  - Considerations: Infrastructure management, scaling responsibility
  - Flow: Cloud Logging → Log Router → Pub/Sub Topic → Custom Application → LogScale
- Cloud Storage + Batch Processing: For historical data or batch analytics
  - Best for: Archival, compliance, batch processing, cost optimization
  - Advantages: Lower costs, long-term storage, batch processing efficiency
  - Considerations: Higher latency, batch processing complexity
  - Flow: Cloud Logging → Log Router → Cloud Storage → Batch Job → LogScale
Identify log sources to collect:
- Platform logs: Automatically collected from GCP services
  - Audit logs (Admin Activity, Data Access, System Event, Policy Denied)
  - VPC Flow Logs
  - Cloud DNS logs
  - Load Balancer logs
  - Cloud NAT logs
- Component logs: From GCP services and resources
  - Compute Engine VM logs
  - Google Kubernetes Engine (GKE) logs
  - Cloud Run logs
  - Cloud Functions logs
  - App Engine logs
  - Cloud SQL logs
- User-written logs: Custom application logs
  - Application logs via Cloud Logging API
  - Structured logs from applications
  - Custom metrics and events
- Multi-cloud logs: AWS logs via Cloud Logging
  - AWS CloudTrail logs
  - AWS VPC Flow Logs
Understand Cloud Logging resource hierarchy:
- Organization: Top-level container for all GCP resources
- Folders: Organizational units within an organization
- Projects: Individual GCP projects containing resources
- Resources: Individual GCP services and components
- Log routing can be configured at any level in the hierarchy
Assess data volumes and costs:
- Review current Cloud Logging ingestion volumes in GCP Console
- Estimate Pub/Sub message volumes and costs
- Calculate Cloud Function invocation costs (if using Cloud Functions)
- Consider log filtering to reduce unnecessary data and costs
- Plan for data egress costs from GCP to LogScale
Plan filtering and exclusion strategies:
- Identify high-volume, low-value logs to exclude (health checks, debug logs)
- Define inclusion filters for critical security and audit logs
- Plan sampling strategies for high-volume logs
- Consider separate sinks for different log priorities
Plan for multi-project and multi-organization scenarios:
- Determine if you need aggregated logging across multiple projects
- Plan for organization-level log sinks if managing multiple projects
- Consider separate Pub/Sub topics per project or consolidated topics
- Plan tagging strategy to identify log sources in LogScale
Plan network connectivity:
- Verify Cloud Functions can reach LogScale endpoints (public or VPC)
- Configure VPC Service Controls if required for security
- Plan for Private Service Connect if using private connectivity
- Consider Cloud NAT for outbound connectivity from private resources

Step 3 - Configure Google Cloud IAM permissions

Why? Proper IAM configuration ensures secure, least-privilege access for the log routing pipeline. Cloud Functions and other components need specific permissions to read from Pub/Sub, write logs, and interact with GCP services.

Detailed steps:

Create a service account for the Cloud Function:
1. Navigate to IAM & Admin → Service Accounts in the GCP Console
2. Click Create Service Account
3. Provide a descriptive name (for example, logscale-log-forwarder)
4. Add description: Service account for forwarding Cloud Logging data to LogScale
5. Click Create and Continue
Grant required IAM roles to the service account:
- Pub/Sub Subscriber (roles/pubsub.subscriber):
  - Allows the Cloud Function to pull messages from Pub/Sub
  - Required for reading log entries from the Pub/Sub topic
- Logging Log Writer (roles/logging.logWriter) (optional):
  - Allows the Cloud Function to write its own logs to Cloud Logging
  - Useful for monitoring and troubleshooting the function
Grant Cloud Logging permission to publish to Pub/Sub:
1. Identify the Cloud Logging service account:
  - Format: service-[PROJECT_NUMBER]@gcp-sa-logging.iam.gserviceaccount.com
  - Find your project number in GCP Console → Dashboard
2. This permission will be granted when creating the log sink (Step 4)
For organization-level log routing, grant additional permissions:
- Logging Admin (roles/logging.admin) at organization level:
  - Required to create organization-level log sinks
- Organization Administrator (roles/resourcemanager.organizationAdmin):
  - Required to manage organization-level resources
Document the service account and permissions:
- Record the service account email address
- Document the granted roles and their purposes
- Store this information securely for future reference

Step 4 - Create Pub/Sub topic and configure log routing

Why? Pub/Sub acts as a reliable, scalable message queue between Cloud Logging and your ingestion pipeline. Log routing (sinks) direct specific logs from Cloud Logging to the Pub/Sub topic based on filters you define.

Detailed steps:

Create a Pub/Sub topic:
1. Navigate to Pub/Sub → Topics in the GCP Console
2. Click Create Topic
3. Configure the topic:
  - Topic ID: Provide a descriptive name (for example, logscale-cloud-logging)
  - Add a default subscription: Leave unchecked (subscription will be created by Cloud Function)
  - Encryption: Choose Google-managed or customer-managed encryption key
4. Configure advanced settings (optional):
  - Message retention duration: Default 7 days (adjust based on recovery requirements)
  - Message storage policy: Configure regions for data residency requirements
5. Click Create
Create a log sink (log router):
1. Navigate to Logging → Log Router in the GCP Console
2. Click Create Sink
3. Configure sink details:
  - Sink name: Provide a descriptive name (for example, logscale-all-logs)
  - Sink description: Document the purpose (for example, Routes all logs to LogScale via Pub/Sub)
4. Click Next
Select sink destination:
1. Choose Cloud Pub/Sub topic as the sink service
2. Select the Pub/Sub topic created in the previous step
3. Click Next

Configure inclusion filters to select logs:

# Include all logs (use with caution - high volume)
# Leave filter empty or use:
resource.type=*

# Include specific resource types
resource.type="gce_instance"
OR resource.type="k8s_container"
OR resource.type="cloud_function"

# Include audit logs only
logName:"cloudaudit.googleapis.com"

# Include specific severity levels
severity >= ERROR

# Include logs from specific projects
resource.labels.project_id="my-project-id"

# Complex filter example: GKE logs excluding health checks
resource.type="k8s_container"
-httpRequest.requestUrl=~"/healthz"
-httpRequest.requestUrl=~"/readyz"

# Include VPC Flow Logs
resource.type="gce_subnetwork"
logName:"compute.googleapis.com/vpc_flows"

Filters use Cloud Logging query language syntax
More specific filters reduce data volume and costs
Test filters in Logs Explorer before applying to sinks

Click Next

Configure exclusion filters (optional but recommended):

Click Add exclusion

Define exclusion filters to reduce noise and costs:

# Exclude health check logs
httpRequest.requestUrl=~"/healthz"
OR httpRequest.requestUrl=~"/readyz"
OR httpRequest.requestUrl=~"/_ah/health"

# Exclude debug logs
severity="DEBUG"

# Exclude specific user agents
httpRequest.userAgent=~"GoogleHC"
OR httpRequest.userAgent=~"kube-probe"

# Exclude high-volume, low-value logs
resource.type="k8s_container"
resource.labels.container_name="istio-proxy"
severity="INFO"

Provide exclusion name and description
Set exclusion percentage (0-100%) for sampling

Review and create the sink:
1. Review the sink configuration
2. Click Create Sink
3. GCP automatically grants the Cloud Logging service account permission to publish to the Pub/Sub topic
Create additional sinks for different log categories (optional):
- Separate sinks for audit logs, application logs, and infrastructure logs
- Different Pub/Sub topics for different priorities or destinations
- Organization-level sinks for centralized logging across projects
Verify the sink is active:
- Check the sink status in Log Router
- Verify messages are being published to the Pub/Sub topic:
  gcloud pubsub topics list gcloud pubsub topics describe logscale-cloud-logging
- Monitor Pub/Sub metrics in GCP Console for message throughput

Step 5 - Deploy Cloud Function to forward logs to LogScale

Why? The Cloud Function acts as the bridge between Pub/Sub and LogScale, consuming log messages from Pub/Sub, transforming them as needed, and forwarding them to LogScale's ingestion API. This serverless approach provides automatic scaling and minimal operational overhead.

Detailed steps:

Prepare the Cloud Function code:

Create a directory for the function code:

mkdir logscale-forwarder
cd logscale-forwarder

Create main.py with the following code:

import base64
import json
import os
import requests
from google.cloud import logging

# Configuration from environment variables
LOGSCALE_URL = os.environ.get('LOGSCALE_URL', 'https://cloud.humio.com/api/v1/ingest/humio-structured')
LOGSCALE_TOKEN = os.environ['LOGSCALE_TOKEN']
BATCH_SIZE = int(os.environ.get('BATCH_SIZE', '100'))

def forward_to_logscale(event, context):
    """
    Cloud Function triggered by Pub/Sub to forward logs to LogScale.
    
    Args:
        event (dict): Event payload containing Pub/Sub message
        context (google.cloud.functions.Context): Metadata for the event
    """
    
    # Decode Pub/Sub message
    if 'data' in event:
        message_data = base64.b64decode(event['data']).decode('utf-8')
        log_entry = json.loads(message_data)
    else:
        print('No data in Pub/Sub message')
        return
    
    # Transform log entry for LogScale
    transformed_entry = transform_log_entry(log_entry)
    
    # Send to LogScale
    try:
        send_to_logscale([transformed_entry])
        print(f'Successfully forwarded log entry: {log_entry.get("logName", "unknown")}')
    except Exception as e:
        print(f'Error forwarding to LogScale: {str(e)}')
        raise  # Raise exception to trigger retry

def transform_log_entry(log_entry):
    """
    Transform GCP log entry to LogScale format.
    
    Args:
        log_entry (dict): GCP Cloud Logging log entry
        
    Returns:
        dict: Transformed log entry for LogScale
    """
    
    # Extract timestamp
    timestamp = log_entry.get('timestamp', log_entry.get('receiveTimestamp'))
    
    # Build LogScale event
    event = {
        'timestamp': timestamp,
        'attributes': {
            'log_name': log_entry.get('logName', ''),
            'severity': log_entry.get('severity', 'DEFAULT'),
            'insert_id': log_entry.get('insertId', ''),
        }
    }
    
    # Add resource information
    if 'resource' in log_entry:
        resource = log_entry['resource']
        event['attributes']['resource_type'] = resource.get('type', '')
        
        # Flatten resource labels
        if 'labels' in resource:
            for key, value in resource['labels'].items():
                event['attributes'][f'resource_{key}'] = value
    
    # Add log-specific fields
    if 'jsonPayload' in log_entry:
        # Structured JSON logs
        event['attributes'].update(flatten_dict(log_entry['jsonPayload']))
    elif 'textPayload' in log_entry:
        # Text logs
        event['attributes']['message'] = log_entry['textPayload']
    elif 'protoPayload' in log_entry:
        # Protocol buffer logs (audit logs)
        event['attributes'].update(flatten_dict(log_entry['protoPayload']))
    
    # Add HTTP request information if present
    if 'httpRequest' in log_entry:
        http_request = log_entry['httpRequest']
        event['attributes']['http_request_method'] = http_request.get('requestMethod', '')
        event['attributes']['http_request_url'] = http_request.get('requestUrl', '')
        event['attributes']['http_status'] = http_request.get('status', '')
        event['attributes']['http_user_agent'] = http_request.get('userAgent', '')
        event['attributes']['http_remote_ip'] = http_request.get('remoteIp', '')
    
    # Add labels
    if 'labels' in log_entry:
        for key, value in log_entry['labels'].items():
            event['attributes'][f'label_{key}'] = value
    
    return event

def flatten_dict(d, parent_key='', sep='_'):
    """
    Flatten nested dictionary.
    
    Args:
        d (dict): Dictionary to flatten
        parent_key (str): Parent key for nested items
        sep (str): Separator for nested keys
        
    Returns:
        dict: Flattened dictionary
    """
    items = []
    for k, v in d.items():
        new_key = f'{parent_key}{sep}{k}' if parent_key else k
        if isinstance(v, dict):
            items.extend(flatten_dict(v, new_key, sep=sep).items())
        elif isinstance(v, list):
            items.append((new_key, json.dumps(v)))
        else:
            items.append((new_key, v))
    return dict(items)

def send_to_logscale(events):
    """
    Send events to LogScale.
    
    Args:
        events (list): List of events to send
    """
    
    headers = {
        'Authorization': f'Bearer {LOGSCALE_TOKEN}',
        'Content-Type': 'application/json'
    }
    
    payload = [{'events': events}]
    
    response = requests.post(
        LOGSCALE_URL,
        headers=headers,
        json=payload,
        timeout=30
    )
    
    response.raise_for_status()
    
    return response

Create requirements.txt:

google-cloud-logging==3.5.0
requests==2.31.0

Deploy the Cloud Function:
1. Using gcloud CLI:
  gcloud functions deploy logscale-forwarder \ --runtime python311 \ --trigger-topic logscale-cloud-logging \ --entry-point forward_to_logscale \ --service-account logscale-log-forwarder@PROJECT_ID.iam.gserviceaccount.com \ --set-env-vars LOGSCALE_URL=https://cloud.humio.com/api/v1/ingest/humio-structured,LOGSCALE_TOKEN=YOUR_INGEST_TOKEN_HERE \ --memory 256MB \ --timeout 60s \ --max-instances 100 \ --region us-central1
2. Using GCP Console:
  1. Navigate to Cloud Functions
  2. Click Create Function
  3. Configure basics:
    Function name: logscale-forwarder
    Region: Choose appropriate region
    Trigger type: Cloud Pub/Sub
    Topic: Select logscale-cloud-logging
  4. Configure runtime settings:
    Memory: 256 MB
    Timeout: 60 seconds
    Max instances: 100
    Service account: logscale-log-forwarder
  5. Add environment variables:
    LOGSCALE_URL: https://cloud.humio.com/api/v1/ingest/humio-structured
    LOGSCALE_TOKEN: YOUR_INGEST_TOKEN_HERE
  6. Click Next
  7. Configure code:
    Runtime: Python 3.11
    Entry point: forward_to_logscale
    Copy the code from main.py into the inline editor
    Copy requirements.txt content
  8. Click Deploy
Verify the Cloud Function deployment:
- Check deployment status in Cloud Functions console
- Verify the function is triggered by the Pub/Sub topic
- Check function logs for any deployment errors:
  gcloud functions logs read logscale-forwarder --region us-central1
Configure advanced Cloud Function settings (optional):
- VPC Connector: For private connectivity to LogScale
- Secrets: Store ingest token in Secret Manager instead of environment variables
- Concurrency: Adjust concurrent executions per instance
- Min instances: Set minimum instances to reduce cold starts

Step 6 - Test and verify

Why? Testing confirms that the integration is working correctly before relying on it in production, ensuring proper log flow from Cloud Logging through Pub/Sub and Cloud Functions to LogScale.

Detailed steps:

Generate test log entries:

Using gcloud CLI:

gcloud logging write test-log "Test log entry from GCP Cloud Logging" \
  --severity=INFO \
  --resource=global

Using Cloud Logging API:

from google.cloud import logging

client = logging.Client()
logger = client.logger('test-logger')

logger.log_text('Test log entry from Python', severity='INFO')
logger.log_struct({
    'message': 'Test structured log',
    'user': 'test-user',
    'action': 'test-action'
}, severity='INFO')

Trigger logs from GCP services (deploy a test Cloud Function, create a GCE instance, etc.)

Verify logs are routed to Pub/Sub:
- Check Pub/Sub topic metrics in GCP Console
- Verify message count is increasing:
  gcloud pubsub topics describe logscale-cloud-logging
- Pull a sample message to verify format:
  gcloud pubsub subscriptions pull logscale-subscription --limit=1
Verify Cloud Function is processing messages:
- Check Cloud Function logs:
  gcloud functions logs read logscale-forwarder --region us-central1 --limit=50
- Look for successful execution messages
- Verify no errors in function logs
- Check function metrics in GCP Console:
  - Invocations per second
  - Execution time
  - Error rate
  - Active instances
Verify data in LogScale:
- Navigate to your repository in LogScale
- Run a query to find recently ingested GCP logs:
  resource_type=* OR log_name=* OR gcp_project_id=*
- Verify events have correct timestamps
- Confirm GCP-specific fields are present:
  - log_name
  - resource_type
  - severity
  - insert_id
  - resource labels (project_id, zone, instance_id, etc.)
- Verify structured log fields are extracted correctly
- Check that HTTP request fields are present (if applicable)
Test different log types:
- Verify audit logs are ingested correctly
- Test GKE container logs
- Test Compute Engine VM logs
- Test Cloud Function logs
- Test VPC Flow Logs (if enabled)
Validate end-to-end latency:
- Generate a log entry with a unique identifier
- Measure time from log generation to availability in LogScale
- Typical latency: 10-60 seconds depending on configuration

Step 7 - Monitoring and maintenance

Why? Ongoing monitoring ensures the reliability, performance, and cost-effectiveness of your Cloud Logging integration, enabling proactive issue detection and continuous optimization of the log ingestion pipeline.

What you should do:

Set up alerts in GCP Cloud Monitoring:
- Cloud Function execution errors exceeding threshold
- Cloud Function execution time exceeding timeout
- Pub/Sub topic message backlog growing
- Pub/Sub subscription oldest unacked message age increasing
- Dead letter queue receiving messages
- Log sink errors or failures
- Cloud Function instance count approaching max
- Unusual drops in log volume
Monitor GCP-specific metrics:
- Cloud Logging ingestion volume and rate
- Log sink routing success rate
- Pub/Sub publish and delivery rates
- Pub/Sub message age and backlog size
- Cloud Function invocation count and rate
- Cloud Function execution time (p50, p95, p99)
- Cloud Function error rate and types
- Cloud Function active instances and scaling behavior
- Network egress from GCP to LogScale
Regularly review and optimize:
- Log sink filters based on actual usage patterns
- Exclusion filters to reduce unnecessary data and costs
- Cloud Function memory and timeout settings
- Cloud Function max instances based on load patterns
- Pub/Sub subscription settings (ack deadline, retry policy)
Implement security best practices:
- Rotate ingest tokens regularly (every 90 days recommended)
- Use Secret Manager for sensitive credentials
- Review and minimize IAM permissions regularly
- Enable VPC Service Controls for sensitive projects
- Audit Cloud Function code changes
- Monitor for unauthorized access to Pub/Sub topics
- Ensure audit logs are always forwarded (never filtered)
Maintain operational documentation:
- Document all log sinks and their purposes
- Maintain runbooks for common troubleshooting scenarios
- Document filter logic and rationale
- Keep inventory of monitored GCP projects and resources
- Document Cloud Function code and transformation logic
- Maintain change logs for configuration modifications

Versions of this Page

Getting Data In

How is Data Impacted?

How to Get Data in

Manage Data Ingest

Package Marketplace

Package Reference

Dashboard Reference

Package Management

Other Integrations

Log Formats

Set up Google Cloud Logging as an ingest method

Enter search term