15:00:35 <yamahata> #startmeeting neutron_northbound
15:00:35 <odl_meetbot> Meeting started Mon Aug 28 15:00:35 2017 UTC.  The chair is yamahata. Information about MeetBot at http://ci.openstack.org/meetbot.html.
15:00:35 <odl_meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:35 <odl_meetbot> The meeting name has been set to 'neutron_northbound'
15:00:47 <yamahata> #chair mpeterson
15:00:47 <odl_meetbot> Current chairs: mpeterson yamahata
15:00:56 <yamahata> #topic agenda bashing and roll call
15:01:00 <yamahata> #info yamahata
15:01:03 <rajivk_> #info rajivk
15:01:04 <jhershbe> #info jhershbe
15:01:04 <mpeterson> #info mpeterson
15:01:17 <yamahata> #chair rajivk jhershbe
15:01:17 <odl_meetbot> Current chairs: jhershbe mpeterson rajivk yamahata
15:01:45 <yamahata> any additional topic?
15:01:47 <jhershbe> for the agenda: I'd like to discuss the carbon gate failures.
15:02:04 <mpeterson> me too, but another aspect
15:02:12 <mpeterson> and also a bug with journal deps calcs
15:02:19 <yamahata> this week is pike release, so we need to settle down final patches.
15:02:54 <mpeterson> yamahata: yes, this could be a priority bug
15:02:58 <jhershbe> yamahata, I think it would be important to nail down the gate issues then, no?
15:03:15 <yamahata> jhershbe: gate issue is priority.
15:03:49 <yamahata> any other topics? rajivk ?
15:04:08 <rajivk_> no
15:04:28 <yamahata> okay
15:04:29 <yamahata> #topic Announcements
15:04:36 <yamahata> This week is pike release
15:04:46 <yamahata> #link https://releases.openstack.org/pike/schedule.html
15:05:40 <yamahata> also odl nitrogen release is coming too
15:05:46 <yamahata> #link https://wiki.opendaylight.org/view/Simultaneous_Release:Nitrogen_Release_Plan
15:05:53 <yamahata> any other announcement?
15:06:06 <mpeterson> no
15:06:14 <rajivk_> no
15:06:20 <yamahata> move on
15:06:21 <yamahata> #topic action items from last meeting
15:06:27 <yamahata> no special action items
15:06:34 <yamahata> #topic Pike/Nitrogen planning
15:06:37 <mpeterson> yamahata: there were :)
15:06:42 <mpeterson> yamahata: it's on the wiki
15:07:44 <yamahata> what's that?
15:08:07 <mpeterson> you asked everyone to review https://review.openstack.org/#/c/493966
15:08:35 <yamahata> Oh yes.
15:08:45 <yamahata> Now we can merge it?
15:09:07 <mpeterson> yes
15:09:22 <mpeterson> I was reviewing the unit tests issues and the rate is above 50% but no clue on what's the issue yet
15:09:25 <yamahata> issued workflow.
15:09:28 <mpeterson> I need your collaboration on that
15:09:58 <mpeterson> I need you to send my way CI logs of tests that failed on py27 or py35 but not the other way around
15:10:05 <yamahata> #topic patches/bugs
15:10:34 <mpeterson> because what I had wasn't enough to start figuring out the problem
15:10:34 <yamahata> then this week we have three major patches/bugs.
15:10:50 <yamahata> journal, tempest CI and unit test failure
15:11:04 <mpeterson> and like I said before
15:11:10 <mpeterson> I'm adding this one for review: https://bugs.launchpad.net/networking-odl/+bug/1713447
15:11:38 <mpeterson> and probably this one as well: https://bugs.launchpad.net/networking-odl/+bug/1713487
15:12:29 <mpeterson> the last one might be related to https://bugs.launchpad.net/networking-odl/+bug/1645551 and https://bugs.launchpad.net/networking-odl/+bug/1708241
15:13:26 <mpeterson> and this review touches upon it and the solution worries me: https://review.openstack.org/#/c/331382/
15:13:53 <yamahata> 1708241 failure happens with specific tempest test cases.
15:14:27 <mpeterson> yamahata: it's present on the rally tests and happen on and off
15:14:34 <yamahata> can you please elaborate on your worry?
15:15:05 <mpeterson> yamahata: for example an UPDATE would fallback to a CREATE
15:15:34 <mpeterson> I've seen in the logs that ports that were there before (or seem to be there before) they dissappear and that would create them again blindly
15:15:46 <mpeterson> masking a possible bug
15:16:02 <mpeterson> a bug that seems to be present, I might say
15:17:12 <mpeterson> yamahata: in general, the patch seems to be "blind" fallbacks that have very specific assumptions which I can't say for sure to be accurate
15:17:20 <jhershbe> I agree on this. It seems very wrong to fall back between update and create and potentially can mask all kinds of issues.
15:17:49 <yamahata> I see. what about ignore 404 on delete?
15:18:06 <yamahata> In router case, it happens.
15:18:14 <yamahata> because on delete on error recovery path.
15:18:46 <mpeterson> yamahata: if it was deleted before because of an action that we did, shouldn't be that also deleted from pending tasks?
15:19:13 <mpeterson> yamahata: I think that then, the journal should be updated to not trigger an action that we know beforehand that will fail
15:20:25 <yamahata> I mean common_db_mixin.safe_creation
15:21:08 <yamahata> On error to create resource, it deletes unconditionally the resource.
15:21:20 <yamahata> so only deletion without creation is possible.
15:21:49 <yamahata> some of tempest tests exercise such execution path with l3 router.
15:22:53 <mpeterson> yamahata: I think we need to find a way to handle it better, but not ignoring the error. It's the least of damaging situations though
15:23:14 <yamahata> I'm not sure you understand the situation.
15:23:47 <yamahata> have you checked/read common_db_mixin.safe_creation ?
15:24:27 <yamahata> anyway I don't think this issue isn't very critical for pike release.
15:24:59 <yamahata> I think we can drop 331382 for pike. what do you think?
15:25:30 <mpeterson> yamahata: agree on that
15:26:45 <yamahata> Okay, I classified bug 1708241 as medium
15:27:33 <yamahata> mpeterson: bug/1713447 is priority, right?
15:27:49 <yamahata> Are you seeing the bug with only unit test?
15:27:57 <yamahata> or with tempest?
15:28:06 <mpeterson> yamahata: mmm but one thing is postponing the review and another one not fixing that bug which seems to be part of the cause of rally fails
15:28:26 <mpeterson> yamahata: re: bug/1713447, no, this is happening in rally as well
15:29:03 <yamahata> I see then it's ciritical or high.
15:29:39 <mpeterson> yamahata: although this one in particular doesn't always cause a FAILURE in the tests
15:30:24 <mpeterson> yamahata: seems to be silently ignored and probably the journal insert just deferred to a later retry
15:31:37 <mpeterson> yamahata: and from a preliminary assesment is caused by a race in the interval between calculating dependencies and inserting the journal entry
15:31:45 <yamahata> It seems retry would solve.
15:32:23 <mpeterson> yamahata: yes, like I said there, I am saying *probably* and *seem*
15:32:50 <yamahata> I see.
15:33:05 <yamahata> do you have any idea to make it progress?
15:33:23 <yamahata> At least we can create experimental patch with retry to see the outcome.
15:33:33 <mpeterson> I do, but I would need your seniority on this probably
15:33:59 <mpeterson> yamahata: we can discuss this off meeting
15:34:06 <mpeterson> to keep the meeting moving forward
15:34:10 <yamahata> okay.
15:34:23 <yamahata> the next one is tempest CI.
15:34:29 <yamahata> jhershbe: you're on stage.
15:34:36 <jhershbe> oh, soryr....
15:34:38 <jhershbe> so...
15:35:03 <jhershbe> fixed one issue which was the br-int being created after set-manager was called...that's merged....but there's still another issue...
15:35:42 <yamahata> that was very nice catch
15:36:13 <jhershbe> which is that there is currently a bug in ODL with pre-existing br-int and provider-networks where a patch port is created to the br-ex bridge...and then erased! ;-)...
15:36:22 <yamahata> For odl log level, do you have any request? we can change default log level to ease debug.
15:36:28 <jhershbe> ...it's going to take a little while to fix that...
15:36:55 <jhershbe> (log level is ok, imho - we can always tweak it for debugging like I've been doing)
15:37:25 <jhershbe> ...my big question is, why are we configuring br-int in devstack at all??? Why  not just allow ODL to create it?
15:37:35 <jhershbe> that should fix the gate
15:38:12 <jhershbe> the patch to create br-int from devstack was 6 weeks ago and before that ODL created it. I believe Manjeet pushed that patch. Do we have any idea why?
15:38:17 <yamahata> If it fixes gate, it's quite fine for devstack not to creat br-int.
15:38:48 <yamahata> It's just because it has been so from legacy netvirt.
15:38:49 <jhershbe> yamahata, GREAT! I will push that
15:39:13 <jhershbe> yamahata, it's not from legacy. It was removed for a long time and then replaced 6 weeks ago
15:39:43 <jhershbe> but whatever
15:39:57 <yamahata> another reason is for ping wit floatingip from host.
15:40:13 <jhershbe> not understanding ^^^
15:40:16 <yamahata> we need to connect br-int between br-int and root netname space.
15:40:58 <jhershbe> wouldn't that be done with a provider network?
15:41:05 <jhershbe> provider-mapping, I mean
15:41:51 <yamahata> Oh, yeah. We can see the result of your future patch.
15:42:13 <jhershbe> yamahata, sure. Probably tomorrow because I need to split right after this meeting
15:43:18 <yamahata> great.
15:43:33 <yamahata> do we have any other bugs/patches?
15:44:07 <rajivk_> For full sync, i have written UT
15:44:15 * mpeterson doesn't have
15:44:28 <yamahata> rajivk_: great
15:44:39 <rajivk_> I will upload the patch, tonight.
15:44:56 <yamahata> It looks like py27, py35 are failing.
15:44:57 <rajivk_> A few are failing, can you give intial comments.
15:45:05 <rajivk_> i will upload new one
15:45:47 <rajivk_> Currently, there seems to be some cleanup issues
15:46:04 <yamahata> It basically looks good.
15:46:05 <rajivk_> i am figuring them out and upload the new patch.
15:46:27 <yamahata> So after fixing failures (and nitpics), I think we can merge it.
15:46:31 <yamahata> and backport it to pike.
15:46:43 <rajivk_> yamahata, Can we backport to ocata as well?
15:47:08 <yamahata> Yes. it's critical bug. you can propose backport.
15:47:49 <rajivk_> yamahata, we should have similar problem with recovery as well
15:48:00 <rajivk_> tomorrow, i will look into that as well.
15:48:01 <yamahata> with master branch?
15:48:09 <jhershbe> yamahata, I pushed that fix for the carbon tempest: https://review.openstack.org/#/c/498506/ (it was a one-liner ;-)
15:48:28 <yamahata> jhershbe: let's see the result.
15:48:37 <jhershbe> yamahata, yup
15:48:43 <jhershbe> gotta drop
15:48:45 <jhershbe> bye
15:48:48 <yamahata> rajivk_: Okay, can you please file bug.
15:48:55 <yamahata> jhershbe: thank you and bye.
15:49:27 <rajivk_> yamahata, okay, i did not test it. If i found similar issue, i will log the bug
15:49:47 <yamahata> what kind of issue? can you share it a bit?
15:50:16 <yamahata> ah, _get_latest_resource ?
15:50:43 <rajivk_> it might have method name different, like we have for full sync
15:51:15 <yamahata> I see. It's very likely.
15:51:37 <rajivk_> https://github.com/openstack/networking-odl/blob/master/networking_odl/journal/recovery.py#L64
15:52:14 <rajivk_> It might be the problem, let me confirm tomorrow and i will proceed accordingly
15:52:22 <yamahata> Cool.
15:52:59 <yamahata> anything else to discuss?
15:53:04 <yamahata> we have 7mins left
15:53:27 <mpeterson> no
15:53:30 <rajivk_> no
15:53:34 <yamahata> #topic open mike
15:53:51 <mpeterson> yamahata: I will probably want to chat with you tomorrow, when do you become available?
15:53:55 <yamahata> we're on final streach for pike release.
15:54:13 <yamahata> let me see...
15:54:37 <yamahata> you're in isreal, right?
15:54:43 <mpeterson> correct
15:54:47 <mpeterson> it's +2
15:54:53 <mpeterson> and you are -8, right?
15:54:56 <yamahata> It's now night there.
15:55:00 <mpeterson> indeed
15:55:06 <mpeterson> 18:55
15:55:22 <mpeterson> not night, but I really need to head back home :)
15:55:50 <yamahata> e.g. PDT 11pm for me works for me...
15:55:55 <yamahata> let me check your time...
15:56:18 <mpeterson> that would be 09am
15:56:35 <mpeterson> it's good for me
15:56:37 <yamahata> does it work for you?
15:56:39 <mpeterson> amazing actually
15:56:47 <mpeterson> I can also do 10pm your time
15:57:01 <mpeterson> of your today, my tomorrow, right?
15:57:05 <yamahata> Right.
15:57:21 <mpeterson> first thing in the morning I'll ping you then
15:57:22 <yamahata> then 10pm is better for me.
15:57:27 <mpeterson> thanks for the stretch :)
15:57:42 <yamahata> anything else?
15:57:45 <mpeterson> if I start earlier than 08am I'll ping you before
15:57:48 <mpeterson> no, thanks
15:57:58 <yamahata> okay, thank you for everyone.
15:58:22 <yamahata> #topic cookies
15:58:26 <yamahata> #endmeeting