Skip to content

Bug: kafka testcontainer does not support adding a network #637

@champialex

Description

@champialex

Describe the bug

KafkaContainer.tc_start only allows containers to have a single IP, so it is not possible to add a network

Changing
listeners = f"PLAINTEXT://{host}:{port},BROKER://$(hostname -i):9092" to
listeners = f"PLAINTEXT://{host}:{port},BROKER://$(hostname -i | cut -d' ' -f1):9092" fixes it

Exception in the container logs:

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://192.168.64.2:33647,BROKER://172.17.0.3 172.21.0.2:9092': Unable to parse BROKER://172.17.0.3 172.21.0.2:9092 to a broker endpoint
	at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:266)
	at kafka.server.KafkaConfig.effectiveAdvertisedListeners(KafkaConfig.scala:2154)
	at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2275)
	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2233)
	at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1604)
	at kafka.Kafka$.buildServer(Kafka.scala:72)
	at kafka.Kafka$.main(Kafka.scala:91)
	at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse BROKER://172.17.0.3 172.21.0.2:9092 to a broker endpoint
	at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:58)
	at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$10(CoreUtils.scala:263)
	at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
	at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
	at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37)
	at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:263)
	... 7 more

To Reproduce

    from kafka import KafkaAdminClient
    from testcontainers.kafka import KafkaContainer

    with Network() as network:
        kafka_ctr = KafkaContainer()
        kafka_ctr.with_network(network)
        kafka_ctr.with_network_aliases("kafka")

        with kafka_ctr:
            print("started") # Will not reach here and timeout
            admin_client = KafkaAdminClient(bootstrap_servers=[kafka_ctr.get_bootstrap_server()])
            print(admin_client.describe_cluster())

Runtime environment

[[package]]
name = "testcontainers"
version = "4.7.1"

# Get the operating system information (on a unix os).
$ uname -a
Darwin sydmacg366 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:10 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6031 arm64
# Get the python version.
$ python --version
Python 3.12.2
# Get the docker version and other docker information.
$ docker info

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /Users/adecham/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.28.1
    Path:     /Users/adecham/.docker/cli-plugins/docker-compose

Server:
 Containers: 338
  Running: 1
  Paused: 0
  Stopped: 337
 Images: 24
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
 runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 init version:
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.55-0-virt
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 7.753GiB
 Name: colima
 ID: c74f716c-594e-4e7c-8ae1-f9f71efbd2ac
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions