Ciris

Standard environment variables and system properties for configuring a KPLProducer, via Ciris

Installation

libraryDependencies += "io.github.etspaceman" %% "kinesis4cats-kpl-ciris" % "0.0.28"

Usage

import cats.effect.IO

import kinesis4cats.kpl.ciris.KPLCiris

KPLCiris.kpl[IO]()

Configuration

All of the configuration below is optional.

Environment Variable System Property Default Description
KPL_ADDITIONAL_METRICS_DIMENSIONS kpl.additional.metrics.dimensions Add additional, custom dimensions to the KPL metrics
KPL_CA_CERT_PATH kpl.ca.cert.path Path to a provided cacert
KPL_AGGREGATION_ENABLED kpl.aggregation.enabled true If true, the KPL will aggregate multiple records that share a partition key into a single record.
KPL_AGGREGATION_MAX_COUNT kpl.aggregation.max.count 4294967295 Maximum number of items to pack into an aggregated record. Between 1-9223372036854775807
KPL_AGGREGATION_MAX_SIZE kpl.aggregation.max.size 51200 Maximum number of bytes to pack into an aggregated Kinesis record. Between 64-1048576
KPL_CLOUDWATCH_ENDPOINT kpl.cloudwatch.endpoint Custom endpoint for Cloudwatch interactions. Must accept TLS.
KPL_CLOUDWATCH_PORT kpl.cloudwatch.port 443 Port for Cloudwatch interactions. Between 1-65535
KPL_COLLECTION_MAX_COUNT kpl.collection.max.count 500 Maximum number of items to pack into an PutRecords request. Between 1-500
KPL_COLLECTION_MAX_SIZE kpl.collection.max.size 5242880 Maximum amount of data to send with a PutRecords request. Between 52224-9223372036854775807
KPL_CONNECT_TIMEOUT kpl.connect.timeout 6 seconds Timeout for establishing TLS connections. Between 100 ms and 5 minutes
KPL_CREDENTIALS_REFRESH_DELAY kpl.credentials.refresh.delay 5 seconds How often to refresh credentials. Between 1 ms and 5 minutes
KPL_ENABLE_CORE_DUMPS kpl.enable.core.dumps false If set to true, the KPL native process will attempt to raise its own core file size soft limit to 128MB, or the hard limit, whichever is lower. No impact on Windows.
KPL_FAIL_IF_THROTTLED kpl.fail.if.throttled false If true, throttled puts are not retried.
KPL_KINESIS_ENDPOINT kpl.kinesis.endpoint Custom endpoint for Kinesis interactions. Must accept TLS.
KPL_KINESIS_PORT kpl.kinesis.port 443 Port for Kinesis interactions. Between 1-65535
KPL_LOG_LEVEL kpl.log.level info Minimum level of logs. Valid values: trace, debug, info, warning, error
KPL_MAX_CONNECTIONS kpl.max.connections 24 Maximum number of connections to open to the backend. HTTP requests are sent in parallel over multiple connections. Between 1-256
KPL_METRICS_GRANULARITY kpl.metrics.granularity shard Controls the granularity of metrics that are uploaded to CloudWatch. Valid values: global, stream, shard
KPL_METRICS_LEVEL kpl.metrics.level detailed Controls the number of metrics that are uploaded to CloudWatch. Valid values: none, summary, detailed
KPL_METRICS_NAMESPACE kpl.metrics.namespace KinesisProducerLibrary The namespace to upload metrics under
KPL_METRICS_UPLOAD_DELAY kpl.metrics.upload.delay 60 seconds Delay between each metrics upload. Between 1 ms and 60 seconds
KPL_MIN_CONNECTIONS kpl.min.connections 1 Minimum number of connections to open to the backend. Between 1-16
KPL_RATE_LIMIT kpl.rate.limit 150 Limits the maximum allowed put rate for a shard, as a percentage of the backend limits. Between 1-9223372036854775807
KPL_RECORD_MAX_BUFFERED_TIME kpl.record.max.buffered.time 100ms Maximum amount of time a record may spend being buffered before it gets sent. Between 0ms and 292271023 years
KPL_RECORD_TTL kpl.record.ttl 30 seconds Set a time-to-live on records. Records that do not get successfully put within the limit are failed. Between 100ms and 292271023 years
KPL_AWS_REGION kpl.aws.region AWS_REGION/aws.region value Which region to send records to, e.g. us-east-1. Defaults to the AWS_REGION/aws.region values if not set. These values would be set by an EC2 if deployed in AWS.
KPL_REQUEST_TIMEOUT kpl.request.timeout 6 seconds The maximum total time elapsed between when we begin a HTTP request and eceiving all of the response. Between 100ms and 10 minutes
KPL_TEMP_DIRECTORY kpl.temp.directory The maximum total time elapsed between when we begin a HTTP request and eceiving all of the response. Between 100ms and 10 minutes
KPL_VERIFY_CERTIFICATE kpl.verify.certificate true If true, verifies SSL certificates.
KPL_PROXY_HOST kpl.proxy.host If you have users going through a proxy, set the host here.
KPL_PROXY_PORT kpl.proxy.port 443 If you have users going through a proxy, set the port here. Between 1-65535
KPL_PROXY_USER_NAME kpl.proxy.user.name If you have users going through a proxy, set the user-name here.
KPL_PROXY_PASSWORD kpl.proxy.password If you have users going through a proxy, set the password here.
KPL_THREADING_MODEL kpl.threading.model PER_REQUEST Sets the threading model that the native process will use. Valid values are PER_REQUEST, POOLED
KPL_THREAD_POOL_SIZE kpl.thread.pool.size Sets the maximum number of threads that the native process' thread pool will be configured with. Must be a non-negative number.
KPL_USER_RECORD_TIMEOUT kpl.user.record.timeout Set the value when the user submitted records will be timed out at the Java layer. Valid values are durations (e.g. 1 second)
KPL_GRACEFUL_SHUTDOWN_FLUSH_ATTEMPTS kpl.graceful.shutdown.flush.attempts 5 Set the amount of times that the KPL will perform a flush() during its graceful shutdown process.
KPL_GRACEFUL_SHUTDOWN_FLUSH_INTERVAL kpl.graceful.shutdown.flush.attempts 500 millis Set the duration between flush() commands during its graceful shutdown process.

Glue

The KPL can be configured to leverage the AWS Glue schema registry. There are some available configuration values for this.

There is no good documentation on these configuration values, so descriptions are sparse. If you are aware as to what each of these values means, please contribute a description!

Environment Variable System Property Required Default Description
KPL_GLUE_ENABLE kpl.glue.enable No false If true, configures the KPL with Glue. All required variables in this section must be set if true, otherwise none need to be set.
KPL_GLUE_REGION kpl.glue.region Yes AWS Region
KPL_GLUE_COMPRESSION_TYPE kpl.glue.compression.type No NONE If not NONE, compresses data on production. Available values are NONE, ZLIB
KPL_GLUE_ENDPOINT kpl.glue.endpoint No Configure a custom endpoint.
KPL_GLUE_TTL kpl.glue.ttl No
KPL_GLUE_CACHE_SIZE kpl.glue.cache.size No
KPL_GLUE_AVRO_RECORD_TYPE kpl.glue.avro.record.type No Supported format for Avro data. Avaialble values are UNKNOWN, SPECIFIC_RECORD, GENERIC_RECORD
KPL_GLUE_PROTOBUF_MESSAGE_TYPE kpl.glue.protobuf.message.type No Supported format for Protobuf data. Available values are UNKNOWN, POJO, DYNAMIC_MESSAGE
KPL_GLUE_REGISTRY_NAME kpl.glue.registry.name No Name of glue registry
KPL_GLUE_COMPATIBILITY kpl.glue.compatibility No Compatibility of schema evolution for glue. Available values are NONE, DISABLED, BACKWARD, BACKWARD_ALL, FORWARD, FORWARD_ALL, FULL, FULL_ALL
KPL_GLUE_DESCRIPTION kpl.glue.description No
KPL_GLUE_SCHEMA_AUTO_REGISTRATION_ENABLED kpl.glue.schema.auto.registration.enabled No false
KPL_GLUE_TAGS kpl.glue.tags No Values shoule be in format key:value,key2:value2
KPL_GLUE_METADATA kpl.glue.metadata No Values shoule be in format key:value,key2:value2
KPL_GLUE_SECONDARY_DESERIALIZER kpl.glue.secondary.deserializer No
KPL_GLUE_USER_AGENT_APP kpl.glue.user.agent.app No

Duration

Some of these environment variables are loaded as a Scala Duration, and are parsed through its apply method.

For example, if you wanted to set the duration to a day, you could use: - 1d - 1 day - 24 hours - 24h