proxy.golang.org : github.com/pschou/go-flowfile
Go-flowfile is a light weight connection handling tool for sending and receiving FlowFiles via an HTTP/HTTPS exchange. When the HTTPS method is used, the client MUST also present a valid client certificate. FlowFiles are at the heart of Apache NiFi and its flow-based design. A FlowFile is a data record, which consists of a pointer to its content (payload) and attributes to support the content, that is associated with one or more provenance events. The attributes are key/value pairs that act as the metadata for the FlowFile, such as the FlowFile filename. The content is the actual data or the payload of the file. More info: https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html A FlowFile is a logical notion that correlates a piece of data with a set of Attributes about that data. Such attributes include a FlowFile's unique identifier, as well as its name, size, and any number of other flow-specific values. While the contents and attributes of a FlowFile can change, the FlowFile object is immutable. Modifications to a FlowFile are made possible by the ProcessSession. The core attributes for FlowFiles are defined in the org.apache.nifi.flowfile.attributes.CoreAttributes enum. The most common attributes you'll see are filename, path and uuid. The string in quotes is the value of the attribute within the CoreAttributes enum. - Filename ("filename"): The filename of the FlowFile. The filename should not contain any directory structure. - UUID ("uuid"): A unique universally unique identifier (UUID) assigned to this FlowFile. - Path ("path"): The FlowFile's path indicates the relative directory to which a FlowFile belongs and does not contain the filename. - Absolute Path ("absolute.path"): The FlowFile's absolute path indicates the absolute directory to which a FlowFile belongs and does not contain the filename. - Priority ("priority"): A numeric value indicating the FlowFile priority. - MIME Type ("mime.type"): The MIME Type of this FlowFile. - Discard Reason ("discard.reason"): Specifies the reason that a FlowFile is being discarded. - Alternative Identifier ("alternate.identifier"): Indicates an identifier other than the FlowFile's UUID that is known to refer to this FlowFile. More info: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.0.3/bk_developer-guide/content/flowfile.html The `File` entity represents the flowfile. The FlowFile Attributes can be inspected on the wire, before the payload is consumed. This way one can craft switching logic and apply rules before the payload has been brougn in. One can compare this design, of acting on the header alone, to that of a network switch/router in which the Layer 2/3 packet header is inspected before the entire packet is consumed to make a determination on which path the bytes should flow. By this design, of not having, to have the entire file before logic can be done, pipes can be connected and memory use can be kept to a minimum. Standard usage has shown that the overall memory footprint of several utilities doing complex routing of FlowFiles remains around 10-20MB, conversely a standard NiFi package will take a minimum of approximately 800MB and up.
Registry
-
Source
- Documentation
- JSON
purl: pkg:golang/github.com/pschou/go-flowfile
License: BSD-3-Clause
Latest release: about 2 years ago
First release: over 2 years ago
Namespace: github.com/pschou
Dependent repositories: 1
Stars: 0 on GitHub
Forks: 0 on GitHub
See more repository details: repos.ecosyste.ms
Last synced: 3 months ago