{"_id":"55ed7fdadf21af2b009e217d","__v":5,"user":"55e94db887e942230032e40d","githubsync":"","project":"55e94ebde5d0c623003ed868","category":{"_id":"55e94ebee5d0c623003ed86c","pages":["55e94ebfe5d0c623003ed86e","55ed7fdadf21af2b009e217d","55ed86db2e66b621009941a6","55ed879428d7c33700de00e1","55ed88392e66b621009941a9","55ed885cec4c3e3900b75611","55ed88ba2e66b621009941ab","55ed8caba872a80d00acff5d","55ed8ce82e66b621009941c3","560d5df697a0a32f006e9de9","566ff8f33a32d20d00c45b37","5670195e81801f0d00802e1c"],"version":"55e94ebee5d0c623003ed86b","__v":12,"project":"55e94ebde5d0c623003ed868","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-09-04T07:56:46.830Z","from_sync":false,"order":9999,"slug":"documentation","title":"Documentation"},"version":{"_id":"55e94ebee5d0c623003ed86b","project":"55e94ebde5d0c623003ed868","__v":1,"createdAt":"2015-09-04T07:56:46.272Z","releaseDate":"2015-09-04T07:56:46.272Z","categories":["55e94ebee5d0c623003ed86c"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"0.3.0","version":"0.3.0"},"updates":["56cc8a4094c8f00b00b83e75"],"next":{"pages":[],"description":""},"createdAt":"2015-09-07T12:15:22.837Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":4,"body":"Brokers can run in replication mode where multiple JVM processes could act as broker.\n\nOnly a JVM is the *leader broker*, the others will be defined *followers*.\n\nLeadership is obtained using [ZooKeeper ](https://zookeeper.apache.org/) with a simple master election algorithm.\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Architecture\"\n}\n[/block]\nReplication use [BookKeeper](http://bookkeeper.apache.org/) in order to replicate status between brokers.\n\nLeader Broker writes status changes on a set of BK ledgers and followers reads continuously from the same ledgers, replicating every change to their own in memory copy of the status.\n\nMajordodo leverages the fencing features of BK in order to ensure that only on Broker can change the shared \"view\" of the status of the system.\n\n\n\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Checkpoints\"\n}\n[/block]\nBrokers do not share any resource other than BK ledgers and ZK filesystem. In order to be able to delete old ledgers a local checkpoint strategy is needed.\n\nWhen it is time to create a checkpoint a dump of the broker status is saved to the local file system (directory \"snapshot\"), each snapshot file will contain:\n\n  * a dump of the status of every task\n  * a dump of the status of every worker\n\nThe sequence number (ledgerid + seqnumber) of the last entry of the log stream applied to the in memory status at the time of the snapshot.\nThe time between checkpoints is configurable.\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Followers\"\n}\n[/block]\nThe brokers start in *follower* mode and recovers from snapshots directory and BK ledgers. When the recovery is over the leader election will start. While in *follower* mode the broker \"tails\" the log and applies changes to its own copy of the view of the system. When a broker is elected as the leader then it opens the actual ledger (this operation will \"fence\" any other zombie broker...) and it begins to accept incoming requests from clients and from workers.\n\nWhen a new broker joins the cluster it looks on ZK for ledgers, and if the whole set of ledgers is still on BK than it replays the whole logs, if not it connects to the actual leader broker, download a snapshot and recovers from it.","excerpt":"","slug":"broker-replication","type":"basic","title":"Broker Replication"}

Broker Replication


Brokers can run in replication mode where multiple JVM processes could act as broker. Only a JVM is the *leader broker*, the others will be defined *followers*. Leadership is obtained using [ZooKeeper ](https://zookeeper.apache.org/) with a simple master election algorithm. [block:api-header] { "type": "basic", "title": "Architecture" } [/block] Replication use [BookKeeper](http://bookkeeper.apache.org/) in order to replicate status between brokers. Leader Broker writes status changes on a set of BK ledgers and followers reads continuously from the same ledgers, replicating every change to their own in memory copy of the status. Majordodo leverages the fencing features of BK in order to ensure that only on Broker can change the shared "view" of the status of the system. [block:api-header] { "type": "basic", "title": "Checkpoints" } [/block] Brokers do not share any resource other than BK ledgers and ZK filesystem. In order to be able to delete old ledgers a local checkpoint strategy is needed. When it is time to create a checkpoint a dump of the broker status is saved to the local file system (directory "snapshot"), each snapshot file will contain: * a dump of the status of every task * a dump of the status of every worker The sequence number (ledgerid + seqnumber) of the last entry of the log stream applied to the in memory status at the time of the snapshot. The time between checkpoints is configurable. [block:api-header] { "type": "basic", "title": "Followers" } [/block] The brokers start in *follower* mode and recovers from snapshots directory and BK ledgers. When the recovery is over the leader election will start. While in *follower* mode the broker "tails" the log and applies changes to its own copy of the view of the system. When a broker is elected as the leader then it opens the actual ledger (this operation will "fence" any other zombie broker...) and it begins to accept incoming requests from clients and from workers. When a new broker joins the cluster it looks on ZK for ledgers, and if the whole set of ledgers is still on BK than it replays the whole logs, if not it connects to the actual leader broker, download a snapshot and recovers from it.