Stable/fluorine - #13

Change-Id: I510f3d88b49a2d101d31fe350fa01a2fea08dcc5
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>

If the list of shards is completely deleted, we end up jumping
into an empty method -- obviously missing updates. This codepath
is really shared, as DataTreeCandidateNode resolves recursively,
creating apropriate delta for children.

Change-Id: I6a70421a5ca405830a1012798f302b390b7aa2f0
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 3dc4859)

The null check is wrong here, as instanceof will evaluate to a boolean,
which will always result in a non-null reference. Fix this by using
checkArgument().

Change-Id: I9c5c1795227cafafc4393ea86aa464287cd96df2
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 0fdb21f)

These were accidently set really large when we were first
looking at artery a while ago but they blow up with
java.lang.OutOfMemoryError: Direct buffer memory. So
set them to the defaults and comment them out.

Change-Id: I92dc150808601271f879bc3917a036b76af8d914
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit f605b10)

Issue: DOCS-69
Change-Id: Iaf57d73e8d720b910c66cae41a75f10ceefba80f
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Change-Id: Ic6ad9bade1240b08cc112f275ea2d1b4516d5446
Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

This brinsg in the latest fixes.

Change-Id: Ie576ffdd40afccd6a22170525027b9ef8c9a94d8
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

This patch makes sure we do not ignore the datastore type, but
pass it down to MD-SAL's IMDS.

Since we are converting from controller to mdsal LogicalDatastoreType
in multiple places, centralize this via LogicalDatastoreType.toMdsal().

JIRA: MDSAL-370
Change-Id: I5e027fd439e325ff91344de98511e503fc801992
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

Use direct converter method rather than matching through name, which
results in faster mapping.

Change-Id: Ic247ebc5c5704d3eb1953eea4d7f36b1d3d49cb8
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

When a TransactionChain is closed, we need to make sure we throw
the correct (controller) exception instead of re-wrapping the mdsal
one.

Change-Id: I74434115fe14a18ba520fc789fa8a21d932a4a15
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

Since Shard.persistPayload() will shortcut when we do not have
a followers or persistence, we must update metadata before we
notify the cohort of the success, as for commit-immediate
transactions that will result in a purge payload being persisted.
That payload will shortcut, too, and execute on stack, hence
metadata will see purge before commit -- leaving it in inconsistent
leaky state.

Move allMetadataCommittedTransaction() calls from
applyReplicatedPayload() down to just after we modify the data
tree and before we invoke listeners.

Change-Id: I1ed2f62756367982cfff08302c79f1c63a871df2
JIRA: CONTROLLER-1746
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 4a97740)

Change-Id: I1e74db121d23c07fefb8ddfb940f78d790709397
Signed-off-by: Stephen Kitt <skitt@redhat.com>

Touching UntrustedXML during bundle activator should cause it to
bind to the JRE XML provider rather than discovering other members
via a TCCL.

Change-Id: I9c6625be7873ed96d5e292a3165e7f319ac68b56
JIRA: CONTROLLER-1867
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

Change-Id: Ie56b4681b9f699d71bd1dba3500e749b2428928c
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>

It seems we cannot meet our own deadlines in our unit tests, let's
bump them to 500ms instead of 100ms.

Change-Id: Ic0f9f09da3108e150bd3ec2b465d66bb3aa80485
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 1b51c28)

This method exists in the DOM API, but for some reason it was left
out if the Binding API. Introduce it as default methods riding on
the read() method and override them in all known implementations,
so the request is efficiently routed.

JIRA: MDSAL-389
Change-Id: Id56332d6f720396305ac68e40554e62fd17c34fb
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit a6bdbfa)

Remove double start of the javadoc.

Change-Id: I6524d117e3eecaeda77c0e7b40a50053dc944b96
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 21869c9)

Java 8 introduced this method, which bypasses array cloning done
by getParameterTypes(), making it more efficient. Take advantage
of it.

Change-Id: I130a58c8ca667e57ae29c99abdd8066d8ca7dbd7
JIRA: MDSAL-398
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

This adds a trivial package-protected method which holds the
implementation, allowing us to short-circuit some casts.

Change-Id: I41d792c50d87638fc6ba51433cd26bf68980fca2
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit e5bd339)

When we are proxying a DOMMountPoint, we also need to take care
of bridging legacy interfaces on top of MD-SAL interfaces.

This patch refactors DOM{Notification,Rpc}Router to factor out
DOM{Notification,Rpc}Service adapters and creates DOMActionService
adapter. These are then used in DOMMountPointAdapter, which in
turn is used by DOMMountPointServiceImpl.

This deals with the case when an MD-SAL-registered mount point
is accessed via Controller APIs.

JIRA: CONTROLLER-1869
Change-Id: Ide4c636afdbd13e233880ab616aec52e46ea115f
Signed-off-by: Jakub Morvay <jmorvay@frinx.io>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit ddbc3eb)

This patch deals with the case where a mount point is registered
via Controller interfaces and then accessed via MD-SAL interfaces
by providing any missing MD-SAL services via proxies built on top
of their Controller counterparts.

This also adds utility convertors to DOM{DataTree,Rpc}Identifier,
so they are readily available.

JIRA: CONTROLLER-1869
Change-Id: Iece3f1d2a2bba58669114f6f8b8abf89fd188690
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit e1ede09)

ImmutableList.builder() did not pre-size its arrays correctly,
leading to poor performance. That is no longer the case and we can
correctly pre-allocate it -- giving us the ArrayList equivalent in
immutable form, hence skipping copying in
{SchemaPath,YangInstanceIdentifier}.create()

Change-Id: I4a070dca26138f28de69e4a0e1f9b0298ee2312b
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit da9ee8c)

This bumps yantools to latest 2.0.x release.

Change-Id: If73b8eb99219e3beb6624f22fa1528cf282c6b51
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

We really want to retain source iteration order of predicates
being streamed into the system, as they usually match the order
locally.

Refactor NormalizedNodeInputStreamReader to read complete
NodeIdentifierWithPredicate objects, thus allowing us to use
ImmutableOffsetMapTemplate to skip some data checks and copying
around.

MDSAL: YANGTOOLS-917
Change-Id: I162919c7e56109fc98933a4eeae8669b030dc992
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

We know the key statically, hence we can use more efficient maps
to hold our NodeIdentifierWithPredicates. This saves some CPU cycles
as well as produces less garbage.

JIRA: MDSAL-917
Change-Id: I55b89498ab725f8aba85f8e7f1c8bb16f0fe6565
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

This allows us to use ForwardingObject, which will forward
toString() invocations intact, properly identifying registered
listeners.

Change-Id: I486352d41047b20cb5b2735cff911d7d49aaf963
JIRA: CONTROLLER-1874
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

The ShardManager is the aggregation point for shards so we need
to propagate shard leader change events etc to the ClientActorBehavior
to initiate a refresh of the backend info. The ModuleShardBackendResolver
sends a new message, RegisterForShardAvailabilityChanges, to the
ShardManager actor with a Consumer callback that is notified by the
ShardManager when events affecting shard leader availability occur.
The ModuleShardBackendResolver then propagates the event notification
to callbacks registered via a new notifyWhenBackendInfoIsStale method
exposed via the BackendInfoResolver interface, which the
ClientActorBehavior calls.

JIRA: CONTROLLER-1873
Change-Id: I9dbcabf5a75b195c811a22dd522115d329e5dc4b
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 1819f12)

There was a behavior change in Karaf [0] because of which restarting
the container now requires the system property karaf.restart to be
set to true in addition to karaf.restart.jvm property. Update
controller restart logic on quarantined condition for the same.

[0] https://issues.apache.org/jira/browse/KARAF-5179

Change-Id: I7b93eb87f53870efea70f2c9a9b82eeca783aa0b
Signed-off-by: Ajay Lele <ajayslele@gmail.com>
(cherry picked from commit 8db9c58)

JFR shows a lot of re-allocations of the backing byte [] when
serializing SimpleReplicatedLogEntry. We can pretty closely
estimate the total serialized size with some reasonable padding
since the Payload is already serialized. Introduce a
specific akka serializer for this.

Jira: CONTROLLER-1872
Change-Id: I94ac9528657119b5bab60dc2f37eef98f18ca1b0
Signed-off-by: Tom Pantelis <tompantelis@gmail.com>
(cherry picked from commit 4ecb8ec)

This class is not instantiable outside of the package is it does
not leak from NormalizedNodePruner -- therefore it cannot be used
by anyone else. Make sure it's package-private and clean it up
a bit.

JIRA: CONTROLLER-1887
Change-Id: I17911508273d68c2795d38e2156c9407522f55fe
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 8fe9ec7)

Use nullable schema instead, as the caller is package-private and
ends up unpacking the return anyway. This simplifies
NormalizedNodePruner a bit.

JIRA: CONTROLLER-1887
Change-Id: I395dd8266bd7198f66890c4ae3633ecda9281726
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit c4a5948)

DataSchemaContextTree.findChild() provides equivalent functionality,
so use that instead of brewing our own.

JIRA: CONTROLLER-1887
Change-Id: Ia55ac99ce8fa37daf06ce0e7e1615f1bd0a07123
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit b624997)

addBuilder() is called after previous check for seal from all callers,
who do not have any side-effects aside allocating arguments to
addBuilder().

Move the call to checkNotSealed() to addBuilder(), making the code
more straightforward and possibly preventing bugs from creeping in.

JIRA: CONTROLLER-1887
Change-Id: I127385378957deb7f38f1203870a0e52cd630644
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 35fe9a2)

This version of NormalizedNodePruner is reusable across invocations,
and improving CPU and memory efficiency in bulk operations due to
not needing to allocate stack nor lookup DataSchemaContextTree.

JIRA: CONTROLLER-1887
Change-Id: I1798f5f6255cd23702bf3792c1f4f5149f92d208
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 7bef3c9)

Rather than instantiating a pruner for each invocation, keep an
instance around for reuse. In order to facilitate free-standing
modifications, we also keep a reference to the DataSchemaContextTree
bound our current SchemaContext.

JIRA: CONTROLLER-1887
Change-Id: I5d310b12feb1ff46dc2af0e67294a925d48a4957
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 8f30cbc)

Upgraded spotbugs is finding more violations, fix them up.

Change-Id: I8956c5b7a6eef38286f4c2b6fb07ca6a775e6965
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 90f0319)

createCursor() has been deprecated long time ago, use its replacement
instead.

Change-Id: Id8862470ca4ad8c9fdec5c05cbac044c6715e153
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 4db0628)

Simple whitespace cleanup to keep checkstyle happy.

Change-Id: Id51f7b87b632cce92c29cbae8794662fe6feb250
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 4360bec)

During async persistence the actor containment is broken which
results in applyState being called directly from the persistence actor.
This means that the ClientRequestTracker might be missing an entry
for the transaction and the transaction being applied as a foreign candidate
in ShardDataTree.
If this happens there will be a transaction stuck in COMMIT_PENDING state,
blocking any further progress in the shard until its restarted.

JIRA: CONTROLLER-1890
Change-Id: I944b233d13103df08b68baeaf3907c064d1d526e
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

EntityOwnershipShard can break the usual shard rule of only one 3PC
running at one time, causing the retrying write attempting to modify
an already sealed modification.

Change-Id: Iaa56b68a30f9191f7aa14428f39d4d7524aeb09d
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit bb04f62)

Add a stopwatch to LeaderInstallSnapshotState that allows
us to keep track whether a follower took too much time to respond
to InstallSnapshot for whatever reason allowing us to retry
chunks that time out.

JIRA: CONTROLLER-1891
Change-Id: Id443a4cc7a069ad4d9982d537d9a8d82b845ac35
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit f33beec)

In case of read-only transactions it is not sufficient to just
persist the abort payload, we need to also purge the transaction.

Change-Id: I8761040d074743b5a1c5eebbecc6449c30abe47e
JIRA: CONTROLLER-1879
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit bc301a1)

When we are sending abort+purge, make sure we propagate it out
as soon as possible.

Change-Id: I2b63aa20867a8b99b779115f0bb448c0e8ac9a22
JIRA: CONTROLLER-1879
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit 04547138454b730148250f38979436cb839b9892)

In some specific cases we need to customize abort-like handling,
which is already implemented for write-like transactions, but is
not present for read-only transaction.

This patch adds the capability to attach a close() handler and
makes sure AbstractSnapshotBackedTransactionChain takes advantage
of it.

JIRA: CONTROLLER-1879
Change-Id: Ic7027956556b5dd25120ee81613a6151e5dbc501
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

Poisoning entries may involve reaction from their callbacks, which
can attempt to circle back through connections.

Make sure we poison them outside of lock context, so that any
callbacks end up seeing a poisoned connection, but without the lock
being held -- hence the locks can be acquired in-order.

JIRA: CONTROLLER-1893
Change-Id: I26551d052307812e76f3e45024a77dbb83312b17
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit e983d61)

This is to fix failures seen on promoting to Central.

Change-Id: I6204d68c5144437e5af50bb5f2cb29431841d394
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
(cherry picked from commit e856511)

Ask-based protocol does not need tracking of transactions and
histories, as it is not retransmitting requests. It also does
not inform backend about purely-local aborted transactions
(read-write and read-only), which leads to transaction tracking
rangesets having holes where those IDs are used.

This adds the prerequisite handling of disabling from the leader
without adding the actual mechanics.

JIRA: CONTROLLER-1879
Change-Id: I133e7688b492336937f394f0f6c3f080a05a820f
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit 33ade24)
(cherry picked from commit cfdfdee)

When we encounter an ask-based protocol message on a leader,
we persist a payload to make sure no state tracking occurs
for that client.

JIRA: CONTROLLER-1879
Change-Id: I3d12a06ce9e5b65ada5b59bde9f9b5486e5e4ef7
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit c413251)
(cherry picked from commit bdc460d)

With the metadata tracking disabled for ask based protocol
there is no need to track aborts for read only transactions
on the backend.

JIRA: CONTROLLER-1879
Change-Id: I189ae3231bb2f3c0eaa0bbe21a14342446708c5f
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit 7011afc)
(cherry picked from commit f437308)

With metadata disabled for ask based protocol, there is no need
to purge transaction on the backend and pollute the journal.

Tell based protocol has purges handled by the fronted so it
should be safe for us to remove them on the backend for ask
protocol.

JIRA: CONTROLLER-1879
Change-Id: Ia5c9470782bb4d59b78bbaa605ee5d36c7fb1644
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit 4944f54)
(cherry picked from commit 9ba29cd)

This brings in latest upstream upgrades.

Change-Id: I26751df871ffc53cc2e8e153930a6f17fe31f642
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

This brings in latest fixes.

Change-Id: I4edd67eb5547806f2cb6dcc8e4f17814d1d2046e
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

If delete fails, it needs to be logged so change this up.

Change-Id: If4051cb3e5b83dee51f919fc1713cc17f032597e
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit d8ef44c)

When the commitIndex gets updated when we reach consensus, we should
propagate this fact to the sender peer. Otherwise data tree change
listeners on peers may be unnecessarily delayed, as they end up
waiting for heartbeat interval (or non-batched entries) to be propagated
out.

The results in observably-better behaviour between leader and follower,
as the leader is sending out notifications of consensus, without
delayling for heartbeat -- which is reflected in the changes to
LeaderTest.

Furthermore, DelayedMessagesElectionScenarioTest is detecting the change,
as two members may achieve asynchronous consensus -- hence the leader
would be sending two messages (request to persist, commitIndex) to push
the consensus out. This exposes a race in the test, where we do not know
the order between TimeoutNow and AppendEntries. If AppendEntries is
processed by member 2 after TimeoutNow, leader will know about the new
term due to AppendEntriesReply we send out. The cluster will still
converge, but the assert for leader will be different if it manages to
process the message. Fix this by isolating member 2.

JIRA: CONTROLLER-1900
Change-Id: I695ef25c7a4cf8799c9c5e04c2c33fbf3e2f21df
Signed-off-by: Robert Varga <robert.varga@pantheon.tech>

These improve snapshot save/load load times by around 15-20%
on my environment.

Change-Id: I67236f7e97f519993031d462c947588004857c33
Signed-off-by: Tomas Cere <tomas.cere@pantheon.tech>
(cherry picked from commit aec4fef)

Change-Id: Ia0b8f84e2bab482a54896d680e8aea90dc738c5b
Signed-off-by: jenkins-releng <jenkins-releng@opendaylight.org>