What are Data Sources?
A data source is a logical bucket within a repository where LogScale physically groups and stores related events together. LogScale automatically decides which bucket an event belongs to based on its tag values. Events that share the same tag values are stored together in the same data source.
The tagging of events in LogScale is fundamentally about query efficiency. When you run a query, LogScale does not need to scan everything โ it can identify which data sources are relevant and skip the rest entirely. The more precisely your tags describe your data, the more LogScale can eliminate irrelevant data before it even starts searching. Organizing related events together enables faster query performance, similar to how organized filing systems enable faster document retrieval than unsorted collections.
Choosing the Right Tags
The organizing power of data sources depends on balance. Too few tags and your buckets are too broad to be useful. Too many unique tag values and the system fractures into an unmanageable sprawl.
This balance hinges on cardinality. If you tag on a field with a high
number of unique values โ such as a user ID
or session ID โ LogScale creates a
separate data source for every single value. Instead of a small number of
well-populated buckets that queries can efficiently navigate, you end up
with thousands or millions of nearly empty ones.
At that point, the organizational system works against you. More data sources means more overhead, more memory pressure, and ultimately slower queries โ the opposite of what tags are meant to achieve.
Data Source Strategy
Data sources are largely invisible during normal operation โ they are an internal mechanism, not something you interact with directly. However, they are the reason why decisions made early in your implementation, such as parser design and tag selection, have a lasting impact on how well LogScale performs at scale.
Configure your tagging strategy before ingesting data. Changing your tagging strategy after data ingestion requires reingesting data.