Sitelet https://web.archive.org/web/20230221050354/https://github.com/kestra-io/kestra/issues/477
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redpanda for local env #477

Open
raphaelauv opened this issue Feb 17, 2022 · 4 comments
Open

Redpanda for local env #477

raphaelauv opened this issue Feb 17, 2022 · 4 comments
Labels
enhancement New feature or request

Comments

@raphaelauv
Copy link

Issue description

Redpanda is faster and smaller for local env, it could be interesting to replace kafka and zookeeper in the develop docker-compose

https://github.com/kestra-io/kestra/blob/develop/docker-compose.yml

version: '3.8'

services:
  kafka-broker:
    image: docker.vectorized.io/vectorized/redpanda:v21.11.4
    ports:
      - "9092:9092"
      - "9081:8081"
      - "9082:8082"
    expose:
      - 29092
      - 8081 # schema-registry
      - 8082 # restproxy
    command:
      - redpanda
      - start
      - --smp
      - '1'
      - --reserve-memory
      - 0M
      - --overprovisioned
      - --node-id
      - '0'
      - --kafka-addr
      - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr
      - PLAINTEXT://kafka-broker:29092,OUTSIDE://localhost:9092


  akhq:
    image: tchiotludo/akhq:0.20.0
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            docker-kafka-server:
              properties:
                bootstrap.servers: "kafka-broker:29092"
              schema-registry:
                url: "http://kafka-broker:8081"
    ports:
      - "9028:8080"
    depends_on:
      - kafka-broker
@tchiotludo
Copy link
Member

I've been thinking that for months also. I've kafka & zookeeper running permanently on my machine.

So I do some research and found that : https://redpanda.com/blog/fast-transactions/: support for transaction is partial for now, Kestra heavily use Kafka transactions. So I'm not really sure it will work properly with Redpanda.

Maybe for local env and most of people, it could be sufficient.

Do you have test on your side ?

@raphaelauv
Copy link
Author

They will apparently soon work on it -> https://github.com/redpanda-data/redpanda/milestone/42

@tchiotludo
Copy link
Member

I've made a quick test :

version: "3.6"

volumes:
  redpanda-data:
    driver: local
  elasticsearch-data:
    driver: local
  kestra-data:
    driver: local

services:
  kafka:
    image: docker.vectorized.io/vectorized/redpanda
    command:
      - redpanda
      - start
      - --smp
      - '1'
      - --reserve-memory
      - 0M
      - --overprovisioned
      - --node-id
      - '0'
      - --kafka-addr
      - PLAINTEXT://0.0.0.0:9092
      - --advertise-kafka-addr
      - PLAINTEXT://kafka:9092

    volumes:
      - redpanda-data:/var/lib/redpanda/data

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    environment:
      discovery.type: single-node
      cluster.routing.allocation.disk.threshold_enabled: "false"
      xpack.security.enabled: "false"
      ES_JAVA_OPTS: "-Xms256m -Xmx256m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

  kestra:
    image: kestra/kestra:develop-full
    command: server standalone
    volumes:
      - kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra:/tmp/kestra
    environment:
      KESTRA_CONFIGURATION: |
        kestra:
          kafka:
            client:
              properties:
                bootstrap.servers: kafka:9092
          elasticsearch:
            client:
              http-hosts: http://elasticsearch:9200
          repository:
            type: elasticsearch
          storage:
            type: local
            local:
              base-path: "/app/storage"
          queue:
            type: kafka
          tasks:
            tmp-dir:
              path: /tmp/kestra/tmp
          url: http://localhost:8080/
    ports:
      - "28080:8080"
    links:
      - kafka

I've an error on the topic creation, don't know why 😞

@tchiotludo tchiotludo added the enhancement New feature or request label Feb 18, 2022
@tchiotludo
Copy link
Member

With the last version (and some minor change in kestra), we have another error :

kestra_1         | 2022-02-23 06:51:21,012 WARN  standalone   i.k.r.k.services.KafkaAdminService Unable to update topic 'kestra_workertask'
kestra_1         | java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidConfigurationException: invalid topic property: min.compaction.lag.ms

that link to this issue : redpanda-data/redpanda#998

We need this one before continune

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
All issues
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tchiotludo @raphaelauv