- Remap inspired `reduce` transform to solve multiline merging · Issue #4258 · vectordotdev/vector
- This issue proposes an idea for a new reduce transform that borrows from the new remap transform. The intent is to solve event merging, as well as other reduce-like problems, through robust reduce ...
This post has been translated by AI.
Post summarized by durumis AI
- Vector is a log collection and processing tool released by DataDog, developed in Rust. It offers easier log transformation compared to OTel and enables declarative management.
- However, the lack of updated official documentation and limited related resources can make using Vector challenging. When deploying with Helm, you need to configure customConfig, handle multiline logs, set data_dir, and manage escape processing, among other things.
- Especially when using Helm, pay attention to escape processing. For architecture considerations, refer to the Topology section in the official documentation.
Today, I'm going to introduce a tool called Vector.
It's a product from DataDog.
It takes on the role that tools like promtail and otel, which we are familiar with, play.
It supports log collection, aggregation, and transformation.
The features are as follows:
1. Written in Rust, not Golang
2. Easier to write log transformation code compared to otel
3. Manageable declaratively
4. Well-written official documentation
While the official website highlights the following as advantages...
I excluded them from the above list because most of these advantages are also touted by other tools.
And, talking about the disadvantages I encountered while using it...
1. Outdated official documentation
2. Difficulty finding related materials
That's about it.
While using it, I didn't experience any major inconveniences, but I was disappointed by the lack of related materials and outdated documentation.
I apologize in advance that I'm writing this based on Helm since I only use Helm... 흑....
Create a simple values.yaml file like this, and
Through the above command, I was able to easily integrate logs from k8s with Loki.
I encountered a few issues while writing the above.
First, when writing customConfig, options that were applied by default are deactivated, so you need to write source, transform, and sinks yourself.
Second, due to the nature of k8s logs, logs are received based on lines, so multiline logs, such as error traces, need to be merged.
While source.file has multiline-related options, k8s doesn't have any.
And, through internal discussions within the Vector team, it was decided that transform.reduce is supported and should be used for this purpose.
Third, you need to set data_dir not only in root.data_dir but also inside custom_config...
If the value is not set here, I often encountered permission errors related to logs... ㅎ
Fourth, I used Helm for deployment, and because of this, there were overlapping grammars, and escaping was sometimes necessary.
The documentation shows it like this "{{ metadata }}", and I pasted it as is, which resulted in an error...
So, when using Helm, you must escape it like this "{{ {{ metadata }} }}"!
Additionally, if you have concerns about the architecture, I recommend referring to the following article.
https://vector.dev/docs/setup/deployment/topologies/