case_clause crash at ra_server:make_rpc_effect()

[shino]

First of all, big thanks to ra team for creating the great library!

---

When running my application including ra, crash by case_clause happened.
I'm not sure this is because of ra bug or the ra usage of my application is wrong 😅

Crash report is as follows. (I replaced application name by <myapp>.)

• Erlang/OTP 25.0
• ra 2.2.0

=CRASH REPORT==== 22-Sep-2022::09:21:30.958181 UTC ===
  crasher:
    initial call: ra_server_proc:init/1
    pid: <0.1338.0>
    registered_name: <myapp>_ra_callback
    exception exit: {'EXIT',
                        {{case_clause,undefined},
                         [{ra_server,make_rpc_effect,4,
                              [{file,
                                   "/<myapp>/_build/default/lib/ra/src/ra_server.erl"},
                               {line,1677}]},
                          {ra_server,'-make_pipelined_rpc_effects/3-fun-0-',
                              8,
                              [{file,
                                   "/<myapp>/_build/default/lib/ra/src/ra_server.erl"},
                               {line,1633}]},
                          {maps,fold_1,3,[{file,"maps.erl"},{line,411}]},
                          {ra_server,handle_leader,2,
                              [{file,
                                   "/<myapp>/_build/default/lib/ra/src/ra_server.erl"},
                               {line,388}]},
                          {ra_server_proc,handle_leader,2,
                              [{file,
                                   "/<myapp>/_build/default/lib/ra/src/ra_server_proc.erl"},
                               {line,1003}]},
                          {ra_server_proc,leader,3,
                              [{file,
                                   "/<myapp>/_build/default/lib/ra/src/ra_server_proc.erl"},
                               {line,467}]},
                          {gen_statem,loop_state_callback,11,
                              [{file,"gen_statem.erl"},{line,1419}]},
                          {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,240}]}]}}
      in function  ra_server_proc:handle_leader/2 (/<myapp>/_build/default/lib/ra/src/ra_server_proc.erl, line 1010)
      in call from ra_server_proc:leader/3 (/<myapp>/_build/default/lib/ra/src/ra_server_proc.erl, line 467)
      in call from gen_statem:loop_state_callback/11 (gen_statem.erl, line 1419)

I tried to reproduce the case but failed. It happened only once so far.

---

Some source code reading...

• The case clause https://github.com/rabbitmq/ra/blob/v2.2.0/src/ra_server.erl#L1677-L1691 does not have
  undefined clause.
• The function ra_log:snapshot_index_term() spec https://github.com/rabbitmq/ra/blob/v2.2.0/src/ra_log.erl#L577
  says it can return undefined.

---

As I said before, there can be a mistake on my application side.
So, if missing clause is by design (e.g. some pre-condition of the function), please close this issue.

Thanks again!

[michaelklishin]

Yes, this must be it: an unhandled undefined. The only tricky aspect about handling these is: what do you do when you get an undefined. In some functions, you can only throw an error with a more sensible message.

[kjnilsson]

@shino do you have any more detail as to what you. were doing when this happened? I understand it is hard to reproduce but this is one of those places in the code that "shouldn't happen" but of course if it does we need to try to understand why

[shino]

@shino do you have any more detail as to what you. were doing when this happened? I understand it is hard to reproduce but this is one of those places in the code that "shouldn't happen" but of course if it does we need to try to understand why

This happened in the process of rolling update in a test cluster.
One (probably very important?) thing I failed to point is that data_dir is cleaned up before restarts (I guess this is bad manner...), i.e.

• stop one node
• remove data_dir
• start the node

The error report associated with the crash report in the description, some words are replaced/snipped (e.g. ):
https://gist.github.com/shino/7bf24473d107b018df545097d44c0d6a

---

P.S.
I found a bug in my app initialization path yesterday, which may be related with this. Now investigating application logs.

[lukebakken]

Hi @shino 👋

[kjnilsson]

remove data_dir

Raft isn't really designed to deal with this case. The assumption is that the storage is reliable and if you do lose the storage for a node (but not the actual node itself) you'd have to remove that member from the raft cluster then re-add it.

[shino]

Hi @lukebakken !!

@kjnilsson

Raft isn't really designed to deal with this case.

Indeed. I will close this issue because of application bug on my side.

Thank you very much for your help!