In below 3 scenarios Replicate holds memory even when not processing data
- Batch tuning memory if consumed by a large batch
- Endpoint (specifically cloud) where memory equivalent to file size set is held. Example: 2 GB for snowflake
- Stream buffers (needed to be bumped when processing high velocity/volume changes) and LOB columns processing
Ask: Task design should be in such a way that Replicate should release the memory for above when not processing any data.
Impact: We have tasks that get high volume/velocity changes during peak hours and as a result are holding ~2GB for each task (20 plus tasks) even when not processing any data. We are growing fast and with this design, we are ending up provisioning servers with more memory even though it’s not needed.
